Oct 31 00:13:51.083867 ip-10-0-134-11 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:13:51.556889 ip-10-0-134-11 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:51.556889 ip-10-0-134-11 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:13:51.556889 ip-10-0-134-11 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:51.556889 ip-10-0-134-11 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:13:51.561862 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561764 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.561862 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561862 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561868 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561873 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561876 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561879 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561882 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561885 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561887 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561890 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561893 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561896 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561899 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561902 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561904 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561907 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561910 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561912 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561915 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561918 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.561940 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561921 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561923 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561926 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561928 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561931 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561933 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561936 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561944 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561947 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561950 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561953 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561956 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561958 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561961 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561964 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561967 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561969 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561972 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561975 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561977 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.562382 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561980 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561982 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561985 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561987 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561990 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561992 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561995 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.561997 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562000 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562002 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562005 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562007 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562010 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562014 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562017 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562020 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562023 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562026 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562029 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.562907 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562032 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562035 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562037 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562040 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562042 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562045 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562047 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562050 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562052 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562055 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562057 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562060 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562062 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562065 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562067 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562070 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562072 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562075 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562079 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.563349 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562084 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562087 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562090 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562092 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562095 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562098 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562100 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562103 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562107 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562109 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562112 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562114 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562117 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562120 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562123 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562125 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562128 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562130 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562133 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562135 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.563814 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562139 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562141 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562144 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562146 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562149 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562152 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562155 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562158 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.562995 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563002 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563005 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563008 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563011 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563014 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563017 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563020 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563023 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563025 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563028 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563030 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.564278 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563034 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563037 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563039 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563042 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563044 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563047 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563049 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563052 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563055 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563057 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563060 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563063 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563066 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563068 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563071 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563074 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563077 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563080 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563082 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563085 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.564772 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563088 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563090 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563093 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563095 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563098 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563100 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563103 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563106 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563109 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563112 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563114 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563117 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563120 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563123 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563125 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563128 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563131 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563133 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563136 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563138 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.565280 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563141 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563144 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563146 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563149 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563152 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563154 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563157 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563160 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563162 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563164 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563167 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563169 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563172 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563174 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563177 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563180 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563182 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563185 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563188 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.565767 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563191 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563194 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563197 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563246 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563250 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563252 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563255 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563257 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563260 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563262 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563266 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563270 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563273 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563277 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563282 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563285 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563288 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563291 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563294 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.566222 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563297 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563300 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563303 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563305 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563308 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563310 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563313 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563315 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563318 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563320 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563323 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563327 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563329 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563332 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563335 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563338 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563846 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563851 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563854 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.566702 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563856 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563859 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563862 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563865 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563867 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563869 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563872 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563875 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563878 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563880 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563883 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563886 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563888 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563893 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563896 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563900 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563902 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563905 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563908 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.567149 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563911 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563913 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563916 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563918 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563921 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563924 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563926 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563929 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563932 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563934 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563937 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563940 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563942 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563945 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563947 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563950 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563952 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563955 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563957 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563960 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.567617 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563963 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563965 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563968 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563970 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563973 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563975 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563978 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563985 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563987 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563990 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563992 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563995 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.563998 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564000 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564003 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564005 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564008 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564010 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564013 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564015 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.568088 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564017 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564020 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564023 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564025 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564028 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564030 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564033 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564035 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564038 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564042 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564045 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564047 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564050 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564052 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564054 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564058 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564061 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564064 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564068 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.568566 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564073 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564077 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564079 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564082 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564085 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564088 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564090 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564093 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564095 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564098 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564100 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564103 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564105 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564108 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564111 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564113 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564116 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564119 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564121 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.569018 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564124 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564127 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564131 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564135 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564138 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.564140 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565019 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565030 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565037 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565042 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565046 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565050 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565054 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565060 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565064 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565067 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565070 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565074 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565077 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565080 2571 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565084 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565087 2571 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565091 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:13:51.569469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565094 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565100 2571 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565103 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565107 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565110 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565113 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565118 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565121 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565125 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565128 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565132 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565135 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565139 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565143 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565145 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565150 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565153 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565156 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565159 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565163 2571 flags.go:64] FLAG: --enable-server="true" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565166 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565170 2571 flags.go:64] FLAG: --event-burst="100" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565173 2571 flags.go:64] FLAG: --event-qps="50" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565177 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:13:51.570012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565180 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565183 2571 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565187 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565190 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565194 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565197 2571 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565201 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565204 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565207 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565210 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565213 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565216 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565219 2571 flags.go:64] FLAG: --feature-gates="" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565226 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565230 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565234 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565238 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565241 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565244 2571 flags.go:64] FLAG: --help="false" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565247 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565253 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565256 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565259 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:13:51.570701 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565263 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565267 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565270 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565273 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565276 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565279 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565282 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565286 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565289 2571 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565292 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565295 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565298 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565301 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565304 2571 flags.go:64] FLAG: --lock-file="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565306 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565310 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565313 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565319 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565322 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565325 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565328 2571 flags.go:64] FLAG: --logging-format="text" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565331 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565334 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565337 2571 flags.go:64] FLAG: --manifest-url="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565340 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:13:51.571250 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565346 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565349 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565354 2571 flags.go:64] FLAG: --max-pods="110" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565357 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565360 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565363 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565366 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565369 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565372 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565375 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565383 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565386 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565389 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565392 2571 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565396 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565399 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565402 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565405 2571 flags.go:64] FLAG: --port="10250" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565408 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565411 2571 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ddfff009df4ce2f9" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565414 2571 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565417 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565420 2571 flags.go:64] FLAG: --register-node="true" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565422 2571 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565426 2571 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565429 2571 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:13:51.571859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565432 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565435 2571 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565438 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565442 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565445 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565447 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565450 2571 flags.go:64] FLAG: --runonce="false" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565454 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565457 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565460 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565463 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565466 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565469 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565473 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565476 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565479 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565482 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565485 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565488 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565492 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565500 2571 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565504 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565512 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565517 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565521 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:13:51.572507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565541 2571 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565544 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565547 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565550 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565553 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565556 2571 flags.go:64] FLAG: --v="2" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565561 2571 flags.go:64] FLAG: --version="false" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565565 2571 flags.go:64] FLAG: --vmodule="" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565569 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.565573 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565770 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565787 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565791 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565795 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565797 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565800 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565803 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565807 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565810 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565813 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565816 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565819 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.573115 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565821 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565824 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565827 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565830 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565833 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565835 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565838 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565841 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565844 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565847 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565850 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565852 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565855 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565858 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565863 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565866 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565869 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565871 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565874 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565876 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.573662 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565879 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565882 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565886 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565888 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565891 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565894 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565897 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565899 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565902 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565905 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565909 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565913 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565915 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565919 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565922 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565924 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565927 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565929 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565932 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.574186 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565935 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565938 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565940 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565943 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565947 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565951 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565954 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565960 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565963 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565966 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565969 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565971 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565974 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565976 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565979 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565982 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565985 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565987 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565990 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.574712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565993 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565996 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.565998 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566001 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566004 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566006 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566009 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566012 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566015 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566018 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566021 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566023 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566026 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566029 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566031 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566033 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566036 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566039 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566041 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566058 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.575174 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566063 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566067 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566070 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566072 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566075 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566078 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566081 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566084 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566086 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566089 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566093 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566095 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566098 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566101 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566104 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566106 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.575674 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.566138 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566181 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566185 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566189 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566192 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566195 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566197 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566201 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566203 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566206 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566209 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566212 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566214 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566217 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566220 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566223 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566226 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566230 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566233 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566235 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.576055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566238 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566240 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566243 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566246 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566249 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566251 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566255 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566258 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566260 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566263 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566266 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566268 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566271 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566273 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566276 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566279 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566281 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566284 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566286 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.576521 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566289 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566292 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566294 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566297 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566300 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566303 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566305 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566310 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566312 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566315 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566319 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566322 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566324 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566327 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566330 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566333 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566335 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566338 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566342 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.577096 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566345 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566349 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566352 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566355 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566357 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566360 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566362 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566365 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566367 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566370 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566373 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566375 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566380 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566383 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566386 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566389 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566392 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566395 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566397 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.577569 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566400 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566404 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566407 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566409 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566413 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566416 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566419 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566421 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566424 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566427 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566429 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566432 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566435 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566437 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566440 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566444 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566446 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566449 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566452 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.578066 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566455 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566458 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566461 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566463 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566466 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566469 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566472 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566474 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566477 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566480 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.566482 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.573814 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.573832 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.573847 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.573851 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573912 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573917 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573921 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573924 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573927 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.578518 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573930 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573933 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573936 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573938 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573941 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573943 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573946 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573949 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573952 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573955 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573958 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573960 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573964 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573968 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573971 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573973 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573976 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573978 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573981 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.579055 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573983 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573986 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573989 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573991 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573994 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.573998 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574001 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574003 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574008 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574012 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574015 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574017 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574020 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574023 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574025 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574028 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574030 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574033 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574036 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574038 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.579563 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574041 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574044 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574047 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574050 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574053 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574056 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574058 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574061 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574064 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574067 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574069 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574072 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574074 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574077 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574079 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574082 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574085 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574087 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574090 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574093 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.580044 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574095 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574098 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574101 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574104 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574106 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574109 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574112 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574115 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574118 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574120 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574123 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574126 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574128 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574131 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574134 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574137 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574139 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574142 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574144 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.580494 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574147 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574149 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574152 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574155 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574157 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574160 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574163 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574165 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574168 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574171 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574173 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574176 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574179 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574181 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574184 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574187 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574189 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574193 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574195 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.580960 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574198 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574201 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574203 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574206 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.574236 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574382 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574388 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574391 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574394 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574398 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574401 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574404 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574407 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574409 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574412 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574415 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.581462 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574417 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574420 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574422 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574425 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574428 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574430 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574433 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574435 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574438 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574441 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574443 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574446 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574448 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574451 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574453 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574456 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574459 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574462 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574464 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574467 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.581865 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574469 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574472 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574475 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574478 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574480 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574483 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574486 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574489 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574492 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574494 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574497 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574499 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574502 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574504 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574507 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574509 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574512 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574515 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574517 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574520 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.582330 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574523 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574538 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574540 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574543 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574546 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574548 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574551 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574554 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574557 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574559 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574562 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574564 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574567 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574570 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574572 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574575 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574577 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574580 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574583 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574585 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.583006 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574588 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574591 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574594 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574596 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574599 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574602 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574604 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574607 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574610 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574614 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574617 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574620 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574622 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574625 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574627 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574630 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574633 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574638 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574642 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.583759 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574644 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574647 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574650 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574653 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574655 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574658 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574660 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574662 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574665 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574668 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574671 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574673 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574675 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574680 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574683 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.584242 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574687 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.574711 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574752 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574756 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574759 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574762 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574765 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574768 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574770 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574773 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574775 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574778 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574780 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574783 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574786 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574790 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:13:51.584628 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574793 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574797 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574800 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574802 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574805 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574808 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574810 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574813 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574816 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574818 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574821 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574824 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574826 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574829 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574832 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574834 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574837 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574840 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574843 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574845 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:13:51.585035 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574848 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574851 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574853 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574856 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574858 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574861 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574863 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574866 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574868 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574871 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574874 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574877 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574881 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574886 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574889 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574892 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574894 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574897 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574900 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:13:51.585497 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574903 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574906 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574909 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574912 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574914 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574917 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574920 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574922 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574925 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574927 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574930 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574933 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574936 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574938 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574941 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574943 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574946 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574950 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:13:51.585967 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574955 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574960 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574964 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574969 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574973 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574977 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574982 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574986 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574992 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574995 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.574997 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575000 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575002 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575005 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575008 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575010 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575013 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575016 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575018 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:13:51.586495 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575021 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575024 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575027 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575029 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575032 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575034 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575038 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575040 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575043 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575045 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575048 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575051 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575053 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575056 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575058 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:51.575061 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.575122 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.577559 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.579266 2571 server.go:1014] "Starting client certificate rotation" Oct 31 00:13:51.587059 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.579370 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:13:51.587507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.579404 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:13:51.587507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.581367 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:13:51.587596 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.587572 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:13:51.599601 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.599578 2571 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:13:51.603890 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.603873 2571 log.go:25] "Validated CRI v1 image API" Oct 31 00:13:51.604803 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.604775 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:13:51.608749 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.608726 2571 fs.go:135] Filesystem UUIDs: map[3469d953-9944-440d-a4b0-97348d9ac442:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c38f7ac8-59db-45eb-9847-dcf031289c88:/dev/nvme0n1p4] Oct 31 00:13:51.608834 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.608747 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:13:51.613785 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.613661 2571 manager.go:217] Machine: {Timestamp:2026-10-31 00:13:51.613159502 +0000 UTC m=+0.395311260 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099865 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dfaa2cdb046cfe63b59f4b1adef9a SystemUUID:ec2dfaa2-cdb0-46cf-e63b-59f4b1adef9a BootID:06176286-2e65-4aca-815a-04570f6bfbe3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dc:4a:e9:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dc:4a:e9:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:10:2a:f4:55:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:13:51.614410 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.614398 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:13:51.614501 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.614489 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:13:51.616660 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616629 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:13:51.616815 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616669 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-11.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:13:51.616856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616817 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:13:51.616856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616825 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:13:51.616856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616836 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:13:51.616856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.616849 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:13:51.617820 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.617810 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:13:51.621214 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.621198 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:13:51.621335 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.621326 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:13:51.622094 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622084 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:13:51.622129 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622099 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:13:51.622129 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622110 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:13:51.622129 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622120 2571 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:13:51.622129 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622129 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:13:51.622578 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.622566 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:13:51.625284 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.625267 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:13:51.631087 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.631067 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:13:51.632213 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632198 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:13:51.632255 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632239 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:13:51.632456 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632439 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:13:51.632489 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632470 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:13:51.632489 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632484 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:13:51.632558 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632498 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:13:51.632558 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632511 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:13:51.632558 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.632523 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:13:51.633199 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633188 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:13:51.633199 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633199 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:13:51.633255 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633207 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:13:51.633255 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633213 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:13:51.633255 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633229 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:13:51.633255 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.633239 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:13:51.634313 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.634302 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:13:51.634347 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.634316 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:13:51.638615 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.638600 2571 server.go:1263] "Started kubelet" Oct 31 00:13:51.638704 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.638669 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:13:51.638777 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.638703 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:13:51.638874 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.638842 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:13:51.643195 ip-10-0-134-11 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:13:51.643903 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.643786 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:13:51.646201 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.646133 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:13:51.647013 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.646999 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:13:51.650439 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.650412 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:13:51.650439 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.650435 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:13:51.650639 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.650563 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:51.651198 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651171 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:13:51.651396 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651174 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:13:51.651469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651401 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:13:51.651516 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651492 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:13:51.651516 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651499 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:13:51.651618 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651545 2571 factory.go:55] Registering systemd factory Oct 31 00:13:51.651618 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651571 2571 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:13:51.652130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651911 2571 factory.go:153] Registering CRI-O factory Oct 31 00:13:51.652130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651924 2571 factory.go:223] Registration of the crio container factory successfully Oct 31 00:13:51.652130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.651987 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:13:51.652130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.652008 2571 factory.go:103] Registering Raw factory Oct 31 00:13:51.652130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.652022 2571 manager.go:1199] Started watching for new ooms in manager Oct 31 00:13:51.653028 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.653013 2571 manager.go:319] Starting recovery of all containers Oct 31 00:13:51.662860 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.662807 2571 manager.go:324] Recovery completed Oct 31 00:13:51.667519 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.667499 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.669684 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.669650 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.669684 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.669689 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.669862 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.669703 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.670346 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.670332 2571 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:13:51.670346 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.670345 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:13:51.670405 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.670359 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.670405 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.670376 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:13:51.671838 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.671826 2571 policy_none.go:50] "Start" Oct 31 00:13:51.671874 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.671843 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:13:51.671874 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.671853 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:13:51.673200 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.673184 2571 policy_none.go:44] "Start" Oct 31 00:13:51.676329 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.676311 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.676415 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.676342 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.676415 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.676354 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.710346 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.710290 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:13:51.714627 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.714602 2571 manager.go:341] "Starting Device Plugin manager" Oct 31 00:13:51.715349 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.715324 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:13:51.715349 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715352 2571 server.go:87] "Starting device plugin registration server" Oct 31 00:13:51.715698 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715684 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:13:51.715764 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715700 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:13:51.715817 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715797 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:13:51.715917 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715894 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:13:51.715917 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.715907 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:13:51.716354 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.716335 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:13:51.716429 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.716383 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:51.776213 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.776170 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-11.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:13:51.776213 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.776188 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-11.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:13:51.777671 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.776181 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-11.us-west-2.compute.internal.18e374eb8bfcdcf4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-11.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-11.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:13:51.638330612 +0000 UTC m=+0.420482370,LastTimestamp:2026-10-31 00:13:51.638330612 +0000 UTC m=+0.420482370,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-11.us-west-2.compute.internal,}" Oct 31 00:13:51.803607 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.803569 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:13:51.804787 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.804769 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:13:51.804858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.804795 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:13:51.804858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.804815 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:13:51.804858 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.804851 2571 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:13:51.816785 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.816701 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.821512 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.821489 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.821646 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.821546 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.821646 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.821562 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.821646 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.821594 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.827119 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.827091 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:13:51.827119 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.827091 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-11.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 00:13:51.827119 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.827104 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:13:51.827341 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.827072 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-11.us-west-2.compute.internal.18e374eb8ddb19eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-11.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-11.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-11.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:13:51.669672427 +0000 UTC m=+0.451824192,LastTimestamp:2026-10-31 00:13:51.669672427 +0000 UTC m=+0.451824192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-11.us-west-2.compute.internal,}" Oct 31 00:13:51.858012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.857985 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:13:51.858169 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.858057 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j4plv" Oct 31 00:13:51.905151 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.904932 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal"] Oct 31 00:13:51.905360 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.905219 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.906402 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.906381 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.906510 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.906420 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.906510 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.906440 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.908259 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.908245 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.908434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.908416 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.908499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.908458 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.909216 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909199 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.909300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909215 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.909300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909233 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.909300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909240 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.909300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909252 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.909300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.909252 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.910889 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.910871 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.910985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.910906 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:51.912100 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.912080 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:51.912176 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.912118 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:51.912176 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.912135 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:51.936029 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.936004 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:51.936265 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.936241 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.941768 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.941744 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.941892 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.941778 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-11.us-west-2.compute.internal\": node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:51.943250 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:51.943231 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.953300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.953270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.953421 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.953306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:51.953421 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:51.953329 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054582 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054582 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054585 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054813 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054614 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054813 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054662 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054813 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054663 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.054813 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.054663 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.061071 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.061035 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.162216 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.162125 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.240810 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.240774 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.247356 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.247330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:52.262768 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.262738 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.363165 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.363120 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.463560 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.463458 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.564614 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.564577 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.578789 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.578758 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:13:52.579001 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.578972 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:13:52.579069 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.578970 2571 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a076d63db167a4a51863954deecefa98-6bdc7d033aceb6e1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-11.us-west-2.compute.internal.18e374eb8ddb73c8\": read tcp 10.0.134.11:35720->3.23.47.122:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-11.us-west-2.compute.internal.18e374eb8ddb73c8 default 4984 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-11.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-11.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-11.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:13:51 +0000 UTC,LastTimestamp:2026-10-31 00:13:51.906428044 +0000 UTC m=+0.688579809,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-11.us-west-2.compute.internal,}" Oct 31 00:13:52.664938 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.664882 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.765450 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.765383 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.863369 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.863317 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:08:52 +0000 UTC" deadline="2028-08-05 06:53:18.846456066 +0000 UTC" Oct 31 00:13:52.863369 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.863361 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15462h39m25.983100174s" Oct 31 00:13:52.866186 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.866157 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:52.885842 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.885812 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:13:52.886932 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.886914 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:13:52.887064 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.886953 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:13:52.887064 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:52.886970 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:13:52.967021 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:52.966973 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:13:53.007840 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.007801 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.008008 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.007845 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.052420 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.052342 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:53.063268 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.063238 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.127956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.127919 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:13:53.129153 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.129135 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:13:53.194033 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.194000 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:13:53.446858 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:53.446810 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682248b34b0664951bb6ec422a0d79db.slice/crio-d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b WatchSource:0}: Error finding container d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b: Status 404 returned error can't find the container with id d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b Oct 31 00:13:53.447415 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:53.447387 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7e4a92431d081dd5f2682702d947e2.slice/crio-966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0 WatchSource:0}: Error finding container 966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0: Status 404 returned error can't find the container with id 966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0 Oct 31 00:13:53.454140 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.454121 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:13:53.625917 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.625825 2571 apiserver.go:52] "Watching apiserver" Oct 31 00:13:53.653069 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.653037 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:13:53.668316 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.668282 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:13:53.738963 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.738922 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:13:53.739354 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.739328 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4rxkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal","openshift-multus/multus-nz77q","kube-system/global-pull-secret-syncer-t57nz","openshift-cluster-node-tuning-operator/tuned-qh76s","openshift-dns/node-resolver-82m77","openshift-image-registry/node-ca-s6mps","kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-8x6zh","openshift-multus/multus-additional-cni-plugins-ncrxv","openshift-multus/network-metrics-daemon-8nzvb","openshift-network-diagnostics/network-check-target-rzsvb","openshift-network-operator/iptables-alerter-4nxqs"] Oct 31 00:13:53.742148 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.742122 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.743838 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.743815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:53.743944 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.743890 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:13:53.745235 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.745219 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.746890 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.746870 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.748326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.748307 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.748428 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.748362 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:13:53.749795 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.749775 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.751324 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.751299 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.753271 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.753250 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.755084 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.755062 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.756920 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.756901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.758671 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.758649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.760234 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.760213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:53.760332 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.760289 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:13:53.764154 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764130 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764229 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764229 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764198 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764322 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsm6\" (UniqueName: \"kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.764322 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764291 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764397 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764320 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764397 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764461 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764461 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.764530 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:53.764578 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.764578 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764747 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764626 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764747 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qww8g\" (UniqueName: \"kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.764747 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.764935 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764935 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764822 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.764935 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.764935 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8fm\" (UniqueName: \"kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.765075 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765075 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.764972 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.765075 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765016 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765075 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765055 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765079 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765096 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765133 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765162 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765234 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765253 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765282 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdt4b\" (UniqueName: \"kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtbn\" (UniqueName: \"kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765356 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.765499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.765913 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.765724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.799458 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.799428 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v2pb4\"" Oct 31 00:13:53.800171 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.800126 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.800171 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.800151 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.804447 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.804422 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfr46\"" Oct 31 00:13:53.804559 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.804474 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:13:53.804559 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.804492 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.804559 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.804537 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.804788 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.804771 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.805473 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.805459 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:13:53.809199 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.809154 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"682248b34b0664951bb6ec422a0d79db","Type":"ContainerStarted","Data":"d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b"} Oct 31 00:13:53.810273 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.810252 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"cf7e4a92431d081dd5f2682702d947e2","Type":"ContainerStarted","Data":"966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0"} Oct 31 00:13:53.848285 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848252 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:13:53.848415 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848341 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bvv2r\"" Oct 31 00:13:53.848919 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848897 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9nxs\"" Oct 31 00:13:53.848919 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848903 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5jgbw\"" Oct 31 00:13:53.849061 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848938 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bcxbn\"" Oct 31 00:13:53.849061 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.848980 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ds5xr\"" Oct 31 00:13:53.849275 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.849262 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:13:53.849891 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.849877 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:13:53.850061 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.850049 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7hvcc\"" Oct 31 00:13:53.850204 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.850189 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:13:53.850264 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.850250 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z97vc" Oct 31 00:13:53.850300 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.850256 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdpqj\"" Oct 31 00:13:53.852649 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.852636 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:13:53.853792 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.853776 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.853858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.853818 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:13:53.853858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.853843 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.859095 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859079 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:13:53.859859 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859840 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.859911 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859847 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.859911 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859847 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:13:53.859991 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859927 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:13:53.859991 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.859977 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.866680 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.866738 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.866738 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866706 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.866738 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.866847 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtbn\" (UniqueName: \"kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.866847 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866766 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.866847 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.866847 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866832 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.866989 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866847 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.866989 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.866989 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.866967 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867124 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867010 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867124 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867075 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867124 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.867124 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867128 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867151 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867181 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867208 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsm6\" (UniqueName: \"kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867238 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867238 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867291 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867585 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867678 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867708 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.867760 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867764 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867798 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867797 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867842 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867908 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867947 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.867989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868010 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.868281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868112 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868119 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdt4b\" (UniqueName: \"kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868154 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6t2r\" (UniqueName: \"kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868197 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868211 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868276 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868290 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868301 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868328 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868372 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24zg\" (UniqueName: \"kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.869114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868406 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868604 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868611 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868690 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868753 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868757 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868850 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.869858 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868859 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868872 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868787 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868918 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868947 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868975 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qww8g\" (UniqueName: \"kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869002 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.868995 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869031 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869091 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869140 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8fm\" (UniqueName: \"kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-www4q\" (UniqueName: \"kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:53.870434 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.869310 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.869418 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:54.369369633 +0000 UTC m=+3.151521403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869475 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869504 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvd46\" (UniqueName: \"kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869591 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.870956 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.869683 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.872105 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.872087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.872153 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.872130 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.872361 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.872344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:53.902969 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.902879 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:13:53.902969 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.902937 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.903157 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.902983 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.903157 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.903010 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:13:53.903391 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.903374 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:13:53.903449 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.903376 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:13:53.903449 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.903394 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:13:53.903449 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.903411 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:13:53.910219 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.910194 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.930777 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.930737 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:53.930777 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.930775 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:53.930945 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.930791 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:53.930945 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.930884 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:13:54.43086063 +0000 UTC m=+3.213012391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:53.933215 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.933186 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdt4b\" (UniqueName: \"kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:53.933326 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.933253 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsm6\" (UniqueName: \"kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:53.933878 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.933863 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtbn\" (UniqueName: \"kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:13:53.936775 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.936749 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8fm\" (UniqueName: \"kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:53.936865 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.936828 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qww8g\" (UniqueName: \"kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:53.970862 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.970826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.970939 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.970886 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.970939 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.970919 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971025 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.970949 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971025 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.970992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971099 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971099 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971099 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6t2r\" (UniqueName: \"kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971202 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971202 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24zg\" (UniqueName: \"kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971289 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971198 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971289 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971289 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971227 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971289 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971248 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971478 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971291 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971478 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971338 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971478 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971478 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971412 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971478 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971496 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971577 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971609 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971621 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971627 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-www4q\" (UniqueName: \"kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971662 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971689 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.971763 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvd46\" (UniqueName: \"kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971817 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971848 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971885 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971915 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.971970 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972040 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972059 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.972043 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972074 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:53.972172 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:54.472148447 +0000 UTC m=+3.254300208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:53.972370 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972260 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972296 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972331 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972384 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972408 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972583 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972444 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:53.973131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972933 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.973645 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.972942 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:53.973645 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.973230 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:53.974897 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:53.974880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:54.037885 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.037850 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvd46\" (UniqueName: \"kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:54.038040 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.037931 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-www4q\" (UniqueName: \"kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:54.038040 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.037997 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24zg\" (UniqueName: \"kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:54.038040 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.038014 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6t2r\" (UniqueName: \"kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:54.053092 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.053056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82m77" Oct 31 00:13:54.061934 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.061906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:13:54.064288 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.064253 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53fa6db_ce80_464d_a087_dc7f0bad918d.slice/crio-8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2 WatchSource:0}: Error finding container 8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2: Status 404 returned error can't find the container with id 8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2 Oct 31 00:13:54.069999 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.069980 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nz77q" Oct 31 00:13:54.070842 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.070814 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96031c66_2f80_4d17_922c_e168d72ea1a3.slice/crio-fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8 WatchSource:0}: Error finding container fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8: Status 404 returned error can't find the container with id fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8 Oct 31 00:13:54.075229 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.074094 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:13:54.079343 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.079316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:13:54.079463 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.079415 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b11045e_f44e_4c3c_bce6_18ba33933ab4.slice/crio-2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa WatchSource:0}: Error finding container 2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa: Status 404 returned error can't find the container with id 2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa Oct 31 00:13:54.084562 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.084537 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f05275_a5f0_4f35_8f29_fde30538efe4.slice/crio-642c3b395394c67da362bf1d0a2d7407ef37740000d2173074de32eb589e05c1 WatchSource:0}: Error finding container 642c3b395394c67da362bf1d0a2d7407ef37740000d2173074de32eb589e05c1: Status 404 returned error can't find the container with id 642c3b395394c67da362bf1d0a2d7407ef37740000d2173074de32eb589e05c1 Oct 31 00:13:54.085540 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.085517 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:13:54.089347 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.089310 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5108cf0_9ca6_4eea_b8cf_858b778d5299.slice/crio-e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7 WatchSource:0}: Error finding container e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7: Status 404 returned error can't find the container with id e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7 Oct 31 00:13:54.094547 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.094521 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a308b8_65d6_4b6d_a1de_1c7af22ac808.slice/crio-78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68 WatchSource:0}: Error finding container 78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68: Status 404 returned error can't find the container with id 78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68 Oct 31 00:13:54.096139 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.096113 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:13:54.104712 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.104686 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba72de8_84d9_48f4_87b7_49492edd8e48.slice/crio-5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995 WatchSource:0}: Error finding container 5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995: Status 404 returned error can't find the container with id 5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995 Oct 31 00:13:54.112287 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.112259 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:13:54.119276 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.119255 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:13:54.121324 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.121299 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda5793a_19ef_46b3_8438_51d8f53c46f5.slice/crio-bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724 WatchSource:0}: Error finding container bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724: Status 404 returned error can't find the container with id bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724 Oct 31 00:13:54.128117 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:13:54.128088 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58180f2e_78da_44ae_8a57_7ad29c0b2e8d.slice/crio-b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971 WatchSource:0}: Error finding container b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971: Status 404 returned error can't find the container with id b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971 Oct 31 00:13:54.378057 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.377960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:54.378210 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.378148 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:54.378250 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.378213 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.378197134 +0000 UTC m=+4.160348881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:54.479252 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.479202 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:54.479443 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.479277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:54.479443 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479408 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:54.479443 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479422 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:54.479443 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479441 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:54.479617 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479455 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:54.479617 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479482 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.479463273 +0000 UTC m=+4.261615029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:54.479617 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.479502 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:13:55.479492966 +0000 UTC m=+4.261644726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:54.806091 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.806014 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:54.806736 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:54.806124 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:13:54.813011 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.812979 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724"} Oct 31 00:13:54.814269 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.814236 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s6mps" event={"ID":"b5108cf0-9ca6-4eea-b8cf-858b778d5299","Type":"ContainerStarted","Data":"e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7"} Oct 31 00:13:54.815625 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.815594 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qh76s" event={"ID":"16f05275-a5f0-4f35-8f29-fde30538efe4","Type":"ContainerStarted","Data":"642c3b395394c67da362bf1d0a2d7407ef37740000d2173074de32eb589e05c1"} Oct 31 00:13:54.816784 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.816758 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nz77q" event={"ID":"6b11045e-f44e-4c3c-bce6-18ba33933ab4","Type":"ContainerStarted","Data":"2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa"} Oct 31 00:13:54.817983 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.817955 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nxqs" event={"ID":"96031c66-2f80-4d17-922c-e168d72ea1a3","Type":"ContainerStarted","Data":"fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8"} Oct 31 00:13:54.819206 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.819178 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82m77" event={"ID":"a53fa6db-ce80-464d-a087-dc7f0bad918d","Type":"ContainerStarted","Data":"8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2"} Oct 31 00:13:54.820704 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.820506 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerStarted","Data":"b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971"} Oct 31 00:13:54.821951 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.821928 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" event={"ID":"5ba72de8-84d9-48f4-87b7-49492edd8e48","Type":"ContainerStarted","Data":"5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995"} Oct 31 00:13:54.823352 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.823326 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rxkn" event={"ID":"37a308b8-65d6-4b6d-a1de-1c7af22ac808","Type":"ContainerStarted","Data":"78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68"} Oct 31 00:13:54.852690 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.852650 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-08-05 23:56:51.155248104 +0000 UTC" Oct 31 00:13:54.852690 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:54.852683 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15479h42m56.302569156s" Oct 31 00:13:55.387527 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.387450 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:55.387756 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.387651 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:55.387756 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.387716 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:13:57.387697687 +0000 UTC m=+6.169849459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:55.488298 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.488006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:55.488298 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.488111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:55.488298 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488267 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:55.488641 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488385 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:57.488357451 +0000 UTC m=+6.270509222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:55.488641 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488490 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:55.488641 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488504 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:55.488641 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488517 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:55.488641 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.488557 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:13:57.488546371 +0000 UTC m=+6.270698134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:55.810885 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.810807 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:55.811493 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.810934 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:13:55.811493 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.811105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:55.811493 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:55.811202 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:13:55.852911 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.852835 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-07-20 01:53:42.338400246 +0000 UTC" Oct 31 00:13:55.852911 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:55.852902 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15073h39m46.485502847s" Oct 31 00:13:56.806020 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:56.805977 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:56.806225 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:56.806135 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:13:57.409639 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:57.409595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:57.410202 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.409761 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:57.410202 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.409834 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:01.409812647 +0000 UTC m=+10.191964418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:13:57.510622 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:57.510575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:57.510823 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:57.510652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:57.510901 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.510824 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:13:57.510901 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.510848 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:13:57.510901 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.510862 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:57.511148 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.510922 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:14:01.510903992 +0000 UTC m=+10.293055756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:13:57.511507 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.511344 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:57.511507 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.511413 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:01.511394668 +0000 UTC m=+10.293546433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:13:57.808395 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:57.807772 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:57.808395 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.807930 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:13:57.809060 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:57.808644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:57.809060 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:57.808773 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:13:58.805209 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:58.805171 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:13:58.805784 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:58.805333 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:13:59.806207 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.805761 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:13:59.806207 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.805833 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:13:59.806207 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:59.806154 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:13:59.806846 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:13:59.806241 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:13:59.852053 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.852000 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"682248b34b0664951bb6ec422a0d79db","Type":"ContainerStarted","Data":"2c4f608659111d844cc2b1769cfa73f18d149bc0d743558c6cda5280c3e6567e"} Oct 31 00:13:59.861308 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.860509 2571 generic.go:356] "Generic (PLEG): container finished" podID="cf7e4a92431d081dd5f2682702d947e2" containerID="937289d7e4b9f70925d898fcae0cf1d85a453721731477d8fc73b6eb552e5ff6" exitCode=0 Oct 31 00:13:59.861308 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.861228 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"cf7e4a92431d081dd5f2682702d947e2","Type":"ContainerDied","Data":"937289d7e4b9f70925d898fcae0cf1d85a453721731477d8fc73b6eb552e5ff6"} Oct 31 00:13:59.879749 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.878774 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="4719f5cd0769eb58618f7930e20e5b69a4be6f286f76ef7841de0a41d41da138" exitCode=0 Oct 31 00:13:59.879749 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:13:59.878834 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"4719f5cd0769eb58618f7930e20e5b69a4be6f286f76ef7841de0a41d41da138"} Oct 31 00:14:00.116830 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:00.115912 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" podStartSLOduration=7.115890272 podStartE2EDuration="7.115890272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:59.983147301 +0000 UTC m=+8.765299076" watchObservedRunningTime="2026-10-31 00:14:00.115890272 +0000 UTC m=+8.898042050" Oct 31 00:14:00.805336 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:00.805300 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:00.805517 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:00.805426 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:00.884997 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:00.884951 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"cf7e4a92431d081dd5f2682702d947e2","Type":"ContainerStarted","Data":"332de249bbc3e82b055765abfd6dd6e8676c16f5841f5e3effcbef1939be7dc5"} Oct 31 00:14:00.891107 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:00.891070 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rxkn" event={"ID":"37a308b8-65d6-4b6d-a1de-1c7af22ac808","Type":"ContainerStarted","Data":"a6a2cc18eb3467f94048659dcc5a6bc61ece8d5ac731b8586861c9a65707f7ce"} Oct 31 00:14:01.137014 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.136944 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4rxkn" podStartSLOduration=3.543187579 podStartE2EDuration="9.136922082s" totalImagesPullingTime="5.593734503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:01.13616131 +0000 UTC m=+9.918313080" watchObservedRunningTime="2026-10-31 00:14:01.136922082 +0000 UTC m=+9.919073858" Oct 31 00:14:01.137165 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.137111 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" podStartSLOduration=8.137105754 podStartE2EDuration="8.137105754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:01.011112995 +0000 UTC m=+9.793264765" watchObservedRunningTime="2026-10-31 00:14:01.137105754 +0000 UTC m=+9.919257524" Oct 31 00:14:01.449977 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.449931 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:01.450175 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.450157 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:01.450271 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.450227 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:09.450207547 +0000 UTC m=+18.232359317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:01.551240 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.551190 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:01.551418 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.551291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:01.551504 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.551428 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:01.551504 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.551487 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:09.551468259 +0000 UTC m=+18.333620007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:01.552039 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.552019 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:14:01.552128 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.552045 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:14:01.552128 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.552060 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:01.552128 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.552106 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:14:09.552091171 +0000 UTC m=+18.334242923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:01.809761 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.809672 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:01.809940 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.809823 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:01.810279 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.810256 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:01.810400 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:01.810371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:01.906464 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.906406 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s6mps" event={"ID":"b5108cf0-9ca6-4eea-b8cf-858b778d5299","Type":"ContainerStarted","Data":"6ca8b77ea588fec474defb19095133ddc1d0739eaf6b9b35a56a71074dd16ac4"} Oct 31 00:14:01.910130 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:01.910096 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" event={"ID":"5ba72de8-84d9-48f4-87b7-49492edd8e48","Type":"ContainerStarted","Data":"7d96df753050d354ff7461a650993413b063e91616ce1184fec489eb4dea30a7"} Oct 31 00:14:02.805976 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:02.805936 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:02.806178 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:02.806089 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:03.416476 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.416443 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:14:03.729699 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.729592 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:14:03.416467456Z","UUID":"e296a3f8-9952-4174-ba73-cc54c02de25c","Handler":null,"Name":"","Endpoint":""} Oct 31 00:14:03.731996 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.731595 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:14:03.731996 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.731622 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:14:03.807113 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.805541 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:03.807113 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:03.805711 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:03.807113 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.805850 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:03.807113 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:03.805925 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:03.918856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.918819 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" event={"ID":"5ba72de8-84d9-48f4-87b7-49492edd8e48","Type":"ContainerStarted","Data":"a1189790c60351aecbb005d5c9fb38d8b0317fa9de2515ef097d091a324a735c"} Oct 31 00:14:03.920825 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.920796 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nz77q" event={"ID":"6b11045e-f44e-4c3c-bce6-18ba33933ab4","Type":"ContainerStarted","Data":"77ce2474e276fbb49b9732df8aa009b42b66c10a5fc39207790b1031d403456b"} Oct 31 00:14:03.923544 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:03.923155 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82m77" event={"ID":"a53fa6db-ce80-464d-a087-dc7f0bad918d","Type":"ContainerStarted","Data":"45186cfc65a9aaa12af6c314b1a12f65dc3bc25bab975809ba01e4dcc158ae1d"} Oct 31 00:14:04.039754 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.039699 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nz77q" podStartSLOduration=2.74413282 podStartE2EDuration="12.039680891s" totalImagesPullingTime="9.295548071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:04.039547613 +0000 UTC m=+12.821699385" watchObservedRunningTime="2026-10-31 00:14:04.039680891 +0000 UTC m=+12.821832659" Oct 31 00:14:04.040021 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.039982 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s6mps" podStartSLOduration=5.452712127 podStartE2EDuration="12.039973237s" totalImagesPullingTime="6.58726111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:02.03210758 +0000 UTC m=+10.814259349" watchObservedRunningTime="2026-10-31 00:14:04.039973237 +0000 UTC m=+12.822125006" Oct 31 00:14:04.154430 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.154338 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-82m77" podStartSLOduration=2.687743916 podStartE2EDuration="12.154325964s" totalImagesPullingTime="9.466582048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:04.154020983 +0000 UTC m=+12.936172750" watchObservedRunningTime="2026-10-31 00:14:04.154325964 +0000 UTC m=+12.936477731" Oct 31 00:14:04.548804 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.548713 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:04.549402 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.549377 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:04.805387 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:04.805299 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:04.805564 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:04.805445 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:05.805848 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:05.805656 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:05.806287 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:05.805722 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:05.806287 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:05.805950 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:05.806287 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:05.806064 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:05.930144 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:05.930104 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4nxqs" event={"ID":"96031c66-2f80-4d17-922c-e168d72ea1a3","Type":"ContainerStarted","Data":"164f26144bb9ecd1b5e3ced6ed43cbf7904a0dcae0a73c9afb534799d4ae3a21"} Oct 31 00:14:05.932332 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:05.932298 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="3788a1ee0ff24d0735a0bc40b8bda7d599e8d0487f01317323a2235fa9eaf449" exitCode=0 Oct 31 00:14:05.932508 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:05.932347 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"3788a1ee0ff24d0735a0bc40b8bda7d599e8d0487f01317323a2235fa9eaf449"} Oct 31 00:14:06.046247 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:06.046188 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4nxqs" podStartSLOduration=4.297961242 podStartE2EDuration="14.04616773s" totalImagesPullingTime="9.748206488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:06.045691592 +0000 UTC m=+14.827843362" watchObservedRunningTime="2026-10-31 00:14:06.04616773 +0000 UTC m=+14.828319500" Oct 31 00:14:06.805751 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:06.805718 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:06.806012 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:06.805841 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:06.936464 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:06.936423 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" event={"ID":"5ba72de8-84d9-48f4-87b7-49492edd8e48","Type":"ContainerStarted","Data":"3edbbd3270073ae3229d3496a7323fda7bdaf011f3cbe180f37d7635940b263e"} Oct 31 00:14:06.937808 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:06.937782 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qh76s" event={"ID":"16f05275-a5f0-4f35-8f29-fde30538efe4","Type":"ContainerStarted","Data":"7d5bf4fee51c9071c0c34f869a8aae0d2910be0046ea64efdc12d9747c4ce284"} Oct 31 00:14:07.055499 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.055438 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" podStartSLOduration=3.695276683 podStartE2EDuration="15.055421435s" totalImagesPullingTime="11.360144752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:07.054851847 +0000 UTC m=+15.837003632" watchObservedRunningTime="2026-10-31 00:14:07.055421435 +0000 UTC m=+15.837573204" Oct 31 00:14:07.173143 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.173045 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qh76s" podStartSLOduration=3.085622899 podStartE2EDuration="15.173031986s" totalImagesPullingTime="12.087409087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:07.172787422 +0000 UTC m=+15.954939188" watchObservedRunningTime="2026-10-31 00:14:07.173031986 +0000 UTC m=+15.955183754" Oct 31 00:14:07.806205 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.806109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:07.806856 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.806109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:07.806856 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:07.806234 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:07.806856 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:07.806351 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:07.942977 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.942939 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="47b394a64f369a3467d5987bdd997b4bda05afdc8ea6611571e70745cd1e8af6" exitCode=0 Oct 31 00:14:07.943147 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:07.943033 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"47b394a64f369a3467d5987bdd997b4bda05afdc8ea6611571e70745cd1e8af6"} Oct 31 00:14:08.805281 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:08.805070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:08.805465 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:08.805373 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:09.511667 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.511590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:09.512108 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.511741 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:09.512108 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.511811 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret podName:a0f381bf-9848-4fbe-8b12-acd6fb7aedb1 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.511792334 +0000 UTC m=+34.293944079 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret") pod "global-pull-secret-syncer-t57nz" (UID: "a0f381bf-9848-4fbe-8b12-acd6fb7aedb1") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:14:09.612169 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.612127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:09.612339 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.612192 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:09.612339 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612287 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:09.612339 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612306 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:14:09.612339 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612325 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:14:09.612339 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612336 2571 projected.go:196] Error preparing data for projected volume kube-api-access-g28rg for pod openshift-network-diagnostics/network-check-target-rzsvb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:09.612503 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612356 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.612331885 +0000 UTC m=+34.394483631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:14:09.612503 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.612382 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg podName:ce75cffd-10f6-4f30-a133-72f73169075e nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.61236898 +0000 UTC m=+34.394520726 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g28rg" (UniqueName: "kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg") pod "network-check-target-rzsvb" (UID: "ce75cffd-10f6-4f30-a133-72f73169075e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:14:09.806173 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.806078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:09.806334 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.806078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:09.806334 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.806228 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:09.806334 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:09.806293 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:09.950131 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.950080 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="5db41ee62163e63bd4b1823ca9479b8f2a1208347bf38a089f0d1e1f66c0706f" exitCode=0 Oct 31 00:14:09.950287 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:09.950142 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"5db41ee62163e63bd4b1823ca9479b8f2a1208347bf38a089f0d1e1f66c0706f"} Oct 31 00:14:10.805174 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:10.805133 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:10.805627 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:10.805254 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:11.806825 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.806786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:11.807363 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:11.806912 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:11.807363 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.806949 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:11.807363 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:11.807030 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:11.958967 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.958927 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"170c1f7536f01658c2f525547cd06e011a9616086cddf8f7fb260a7d56dcb5ea"} Oct 31 00:14:11.959111 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.959023 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"eca7e2f49d75d71f1eb1d2331abebb2ad9cfdc785b0be00afe1e378adf5aa63e"} Oct 31 00:14:11.959111 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.959075 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"a21448c37c4b45b1a4776aa3510b72c21fae7160d4526d3f54f8f72794f211d8"} Oct 31 00:14:11.959191 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.959120 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"0a8be54e3061b797de2cdf18f818085a93190aea739b6ca330b6caecd419e0d3"} Oct 31 00:14:11.959191 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.959168 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"82295fd4ffea02548e5aedba2d54d01cc224f6d1970d08b6d5b1028826e295f0"} Oct 31 00:14:11.959254 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:11.959212 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"fbd01769f53a8f96ac12a47f6baf7ed34bbdb9dea7a20e649c4ff7075d4850f0"} Oct 31 00:14:12.413430 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.413396 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:12.413586 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.413524 2571 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:14:12.414012 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.413997 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:12.805045 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.805009 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:12.805194 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:12.805109 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:12.963632 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.963593 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="03fbf6855198ce986a6b688c22d5f2cec79cf01da143013dbcc5029df1a69fb8" exitCode=0 Oct 31 00:14:12.964074 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:12.963668 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"03fbf6855198ce986a6b688c22d5f2cec79cf01da143013dbcc5029df1a69fb8"} Oct 31 00:14:13.805849 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:13.805758 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:13.806015 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:13.805873 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:13.806015 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:13.805905 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:13.806015 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:13.805954 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:13.969117 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:13.969082 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"12756cf9c858151980685e662474286218d2b7966ebec86785d0f2545f06eece"} Oct 31 00:14:13.972073 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:13.972043 2571 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="d29e1f39e448d75705817aa5af66a327a24375905859ac0d33a1cbc1282797bb" exitCode=0 Oct 31 00:14:13.972205 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:13.972084 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerDied","Data":"d29e1f39e448d75705817aa5af66a327a24375905859ac0d33a1cbc1282797bb"} Oct 31 00:14:14.805446 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:14.805410 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:14.805724 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:14.805519 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:14.977221 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:14.977191 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" event={"ID":"58180f2e-78da-44ae-8a57-7ad29c0b2e8d","Type":"ContainerStarted","Data":"8ddf07d098d4f1ad6115c8a045ab7048d6e982dd82807611206635ce5557dd58"} Oct 31 00:14:15.099222 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.099116 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ncrxv" podStartSLOduration=7.621088823 podStartE2EDuration="23.099100579s" totalImagesPullingTime="15.478011756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:15.099070626 +0000 UTC m=+23.881222393" watchObservedRunningTime="2026-10-31 00:14:15.099100579 +0000 UTC m=+23.881252338" Oct 31 00:14:15.107381 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.107340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:15.313718 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.313679 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:15.805623 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.805338 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:15.805623 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.805351 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:15.805837 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:15.805657 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:15.805837 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:15.805754 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:15.983412 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:15.983373 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" event={"ID":"fda5793a-19ef-46b3-8438-51d8f53c46f5","Type":"ContainerStarted","Data":"be4919d1af81a00ac61029adf45575e2820fbb1f9e9f2a0c0349d06a736cf354"} Oct 31 00:14:16.261308 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:16.261215 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" podStartSLOduration=7.46071516 podStartE2EDuration="24.261200881s" totalImagesPullingTime="16.800485721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:16.210794985 +0000 UTC m=+24.992946753" watchObservedRunningTime="2026-10-31 00:14:16.261200881 +0000 UTC m=+25.043352648" Oct 31 00:14:16.389291 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:16.389259 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:16.805782 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:16.805750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:16.805959 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:16.805860 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:16.986574 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:16.986523 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:16.987074 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:16.986650 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:17.002588 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:17.002562 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:17.002736 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:17.002696 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:17.805344 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:17.805309 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:17.805510 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:17.805356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:17.805510 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:17.805427 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:17.805638 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:17.805599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:18.136059 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.135975 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t57nz"] Oct 31 00:14:18.136505 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.136112 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:18.136505 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:18.136235 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:18.138806 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.138783 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rzsvb"] Oct 31 00:14:18.138905 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.138872 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:18.138986 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:18.138967 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:18.139344 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.139328 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nzvb"] Oct 31 00:14:18.139406 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.139397 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:18.139478 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:18.139465 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:18.552558 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.552465 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:18.739406 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:18.739370 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:19.805742 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:19.805669 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:19.805742 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:19.805690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:19.805742 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:19.805741 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:19.806277 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:19.805820 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:19.806277 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:19.805925 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:19.806277 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:19.805994 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:21.248871 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:21.248827 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.436469 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:21.436433 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.806225 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:21.806043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:21.806365 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:21.806129 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:21.806365 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:21.806292 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nzvb" podUID="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" Oct 31 00:14:21.806443 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:21.806162 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:21.806443 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:21.806402 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t57nz" podUID="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" Oct 31 00:14:21.806512 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:21.806447 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rzsvb" podUID="ce75cffd-10f6-4f30-a133-72f73169075e" Oct 31 00:14:22.061548 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.061461 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.202700 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.202664 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.350069 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.349993 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.493584 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.493552 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.641390 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.641358 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.785110 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.785078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:22.937181 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:22.937101 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:23.404843 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.404824 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeReady" Oct 31 00:14:23.405196 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.404933 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:14:23.522928 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.522891 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5c5ss"] Oct 31 00:14:23.528231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.528204 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wj8wj"] Oct 31 00:14:23.528366 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.528325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.531736 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.531718 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t25p7"] Oct 31 00:14:23.531826 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.531815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.535336 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.535296 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.579647 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.579618 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c5ss"] Oct 31 00:14:23.583575 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.583546 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:14:23.584974 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.584949 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wj8wj"] Oct 31 00:14:23.589051 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.589033 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74dxc\"" Oct 31 00:14:23.589143 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.589073 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:14:23.600227 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.600197 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t25p7"] Oct 31 00:14:23.608447 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608424 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:14:23.608578 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.608624 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608586 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.608624 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.608748 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.608814 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.608869 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.608920 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lbf\" (UniqueName: \"kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.608976 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.608942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4zm\" (UniqueName: \"kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.609036 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.609021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.609071 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.609045 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.609114 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.609068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx5f\" (UniqueName: \"kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.680898 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.680812 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:14:23.680898 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.680831 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:14:23.680898 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.680855 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:14:23.681093 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.680910 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:14:23.687044 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.687023 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:14:23.687123 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.687091 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hcz6l\"" Oct 31 00:14:23.687123 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.687091 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:14:23.687196 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.687133 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qszwm\"" Oct 31 00:14:23.703945 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.703923 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-42jns"] Oct 31 00:14:23.709482 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709461 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lbf\" (UniqueName: \"kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.709585 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4zm\" (UniqueName: \"kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709585 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709560 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709697 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.709697 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709627 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx5f\" (UniqueName: \"kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.709697 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709837 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.709702 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:23.709837 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.709762 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:24.209745502 +0000 UTC m=+32.991897266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:23.709837 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.709945 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.709954 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.709963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.709985 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.709991 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls podName:a8513b67-029f-448a-ba14-269e06d1f996 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:24.209979202 +0000 UTC m=+32.992130952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls") pod "dns-default-5c5ss" (UID: "a8513b67-029f-448a-ba14-269e06d1f996") : secret "dns-default-metrics-tls" not found Oct 31 00:14:23.710246 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:23.710013 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:24.209998536 +0000 UTC m=+32.992150295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : configmap "kube-rbac-proxy" not found Oct 31 00:14:23.710246 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.710021 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.710246 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.710208 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.710339 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.710282 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.711014 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.710997 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.714751 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.714731 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.763679 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.763644 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-42jns"] Oct 31 00:14:23.774069 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.774035 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lbf\" (UniqueName: \"kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:23.775266 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.775247 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx5f\" (UniqueName: \"kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.775389 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.775368 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4zm\" (UniqueName: \"kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:23.780626 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.780609 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:14:23.780810 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.780792 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:14:23.780855 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.780795 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:14:23.785491 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.785472 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:14:23.785589 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.785516 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r9nst\"" Oct 31 00:14:23.805769 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.805740 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:23.805769 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.805768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:23.805914 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.805741 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:23.811160 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.811138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qx8\" (UniqueName: \"kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.811297 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.811281 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.811356 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.811345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.847189 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.847155 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:23.882557 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.880292 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:14:23.882557 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.880370 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:14:23.885197 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.885019 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zk5sj\"" Oct 31 00:14:23.885197 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.885023 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:14:23.885197 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.885051 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-plmbm\"" Oct 31 00:14:23.885197 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.885166 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:14:23.885834 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.885502 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lg2hb\"" Oct 31 00:14:23.886733 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.886697 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:23.912962 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.912072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.912962 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.912141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.912962 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.912180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qx8\" (UniqueName: \"kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.913657 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.913606 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.916201 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.916034 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:23.976783 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:23.976712 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qx8\" (UniqueName: \"kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:24.026667 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.026633 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:24.081127 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.081042 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:24.215117 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.215077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:24.215274 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.215153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:24.215274 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.215183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:24.215274 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215236 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:14:24.215407 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215292 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:14:24.215407 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215306 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.215290185 +0000 UTC m=+33.997441947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : configmap "kube-rbac-proxy" not found Oct 31 00:14:24.215407 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215333 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls podName:a8513b67-029f-448a-ba14-269e06d1f996 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.215320159 +0000 UTC m=+33.997471930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls") pod "dns-default-5c5ss" (UID: "a8513b67-029f-448a-ba14-269e06d1f996") : secret "dns-default-metrics-tls" not found Oct 31 00:14:24.215407 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215245 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:24.215407 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:24.215379 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:25.215370473 +0000 UTC m=+33.997522222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:24.230249 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:14:24.230222 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3176d964_e0a4_49f3_b835_6b15b55bb911.slice/crio-cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8 WatchSource:0}: Error finding container cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8: Status 404 returned error can't find the container with id cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8 Oct 31 00:14:24.230249 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.230236 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wj8wj"] Oct 31 00:14:24.400865 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.400841 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-42jns"] Oct 31 00:14:24.400974 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:14:24.400953 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6ddd0c_26be_4ef2_a254_791deccb7ae3.slice/crio-7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829 WatchSource:0}: Error finding container 7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829: Status 404 returned error can't find the container with id 7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829 Oct 31 00:14:24.449045 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:24.449012 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.006436 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.006405 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" event={"ID":"0f6ddd0c-26be-4ef2-a254-791deccb7ae3","Type":"ContainerStarted","Data":"7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829"} Oct 31 00:14:25.007424 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.007401 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wj8wj" event={"ID":"3176d964-e0a4-49f3-b835-6b15b55bb911","Type":"ContainerStarted","Data":"cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8"} Oct 31 00:14:25.223231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.223190 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:25.223231 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.223230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:25.223399 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223325 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 00:14:25.223399 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223336 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:14:25.223399 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.223368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:25.223399 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223383 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:27.22336936 +0000 UTC m=+36.005521105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : configmap "kube-rbac-proxy" not found Oct 31 00:14:25.223586 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223442 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls podName:a8513b67-029f-448a-ba14-269e06d1f996 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:27.223423738 +0000 UTC m=+36.005575488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls") pod "dns-default-5c5ss" (UID: "a8513b67-029f-448a-ba14-269e06d1f996") : secret "dns-default-metrics-tls" not found Oct 31 00:14:25.223586 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223460 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:25.223586 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.223495 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:27.223486028 +0000 UTC m=+36.005637799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:25.526912 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.526863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:25.529552 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.529508 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:25.628444 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.628400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:25.628624 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.628488 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:25.628624 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.628557 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:14:25.628624 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:25.628625 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:57.628611001 +0000 UTC m=+66.410762746 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : secret "metrics-daemon-secret" not found Oct 31 00:14:25.630028 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.630007 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:25.631374 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.631356 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:25.744429 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.744396 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.893507 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.893422 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:25.920181 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:25.919938 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:26.024627 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:14:26.016308 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f381bf_9848_4fbe_8b12_acd6fb7aedb1.slice/crio-a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7 WatchSource:0}: Error finding container a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7: Status 404 returned error can't find the container with id a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7 Oct 31 00:14:26.026476 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:26.026449 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t57nz"] Oct 31 00:14:26.321006 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:26.320974 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rzsvb"] Oct 31 00:14:26.349935 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:26.349829 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:26.658500 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:26.658463 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82m77_a53fa6db-ce80-464d-a087-dc7f0bad918d/dns-node-resolver/0.log" Oct 31 00:14:27.018071 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.017990 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t57nz" event={"ID":"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1","Type":"ContainerStarted","Data":"a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7"} Oct 31 00:14:27.019259 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.019233 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rzsvb" event={"ID":"ce75cffd-10f6-4f30-a133-72f73169075e","Type":"ContainerStarted","Data":"fda91ee06c1e2f8018a576b2c3dcf25c740f0f8a686125467a1d5ac4a02bda58"} Oct 31 00:14:27.241185 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.241143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:27.241369 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.241216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:27.241369 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.241238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:27.241489 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:27.241373 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 00:14:27.241489 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:27.241424 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls podName:a8513b67-029f-448a-ba14-269e06d1f996 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:31.241407666 +0000 UTC m=+40.023559412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls") pod "dns-default-5c5ss" (UID: "a8513b67-029f-448a-ba14-269e06d1f996") : secret "dns-default-metrics-tls" not found Oct 31 00:14:27.241489 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:27.241476 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:27.241653 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:27.241499 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:31.24149187 +0000 UTC m=+40.023643615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:27.242054 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.242030 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:27.458942 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.458901 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s6mps_b5108cf0-9ca6-4eea-b8cf-858b778d5299/node-ca/0.log" Oct 31 00:14:27.481416 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:27.481378 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:28.031560 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:28.031416 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" event={"ID":"0f6ddd0c-26be-4ef2-a254-791deccb7ae3","Type":"ContainerStarted","Data":"5ee857e9955f0bfe368e0f8117660bbb273c75c7913e5e365ca907bdd06b18d2"} Oct 31 00:14:28.147408 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:28.147346 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" podStartSLOduration=1.8419809219999999 podStartE2EDuration="5.147325384s" totalImagesPullingTime="3.305344462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:28.14702816 +0000 UTC m=+36.929179930" watchObservedRunningTime="2026-10-31 00:14:28.147325384 +0000 UTC m=+36.929477149" Oct 31 00:14:29.037015 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:29.036980 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wj8wj" event={"ID":"3176d964-e0a4-49f3-b835-6b15b55bb911","Type":"ContainerStarted","Data":"05f04b123f75b8bfa5aceda125b42c5f3f6a284716e68fdbfe5ca962e0e8c799"} Oct 31 00:14:29.203419 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:29.203365 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wj8wj" podStartSLOduration=1.919966861 podStartE2EDuration="6.203349291s" totalImagesPullingTime="4.28338243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:29.152596146 +0000 UTC m=+37.934747916" watchObservedRunningTime="2026-10-31 00:14:29.203349291 +0000 UTC m=+37.985501057" Oct 31 00:14:29.467251 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:29.467218 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.765429 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:29.765348 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.906881 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:29.906838 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:30.087575 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:30.087418 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:30.235125 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:30.235085 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:30.617138 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:30.617103 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.277128 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.277092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:31.277601 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.277168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:31.277601 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:31.277283 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:31.277601 ip-10-0-134-11 kubenswrapper[2571]: E1031 00:14:31.277367 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:39.277344977 +0000 UTC m=+48.059496739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:31.279888 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.279870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:31.344481 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.344405 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:31.386984 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.386920 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.587682 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.587560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.745498 ip-10-0-134-11 kubenswrapper[2571]: W1031 00:14:31.745471 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8513b67_029f_448a_ba14_269e06d1f996.slice/crio-c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873 WatchSource:0}: Error finding container c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873: Status 404 returned error can't find the container with id c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873 Oct 31 00:14:31.745661 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.745643 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5c5ss"] Oct 31 00:14:31.791987 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.790095 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.991287 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:31.991242 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:32.047871 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:32.047778 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c5ss" event={"ID":"a8513b67-029f-448a-ba14-269e06d1f996","Type":"ContainerStarted","Data":"c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873"} Oct 31 00:14:32.049545 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:32.049504 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rzsvb" event={"ID":"ce75cffd-10f6-4f30-a133-72f73169075e","Type":"ContainerStarted","Data":"1078e57d728d5066e00c03b45b49fb1cbf68528134aa6d058022f03aaa3c799f"} Oct 31 00:14:32.049675 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:32.049636 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:32.168542 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:32.168482 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rzsvb" podStartSLOduration=35.457449626 podStartE2EDuration="40.168469199s" totalImagesPullingTime="4.711019573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:32.168237261 +0000 UTC m=+40.950389028" watchObservedRunningTime="2026-10-31 00:14:32.168469199 +0000 UTC m=+40.950620966" Oct 31 00:14:32.191984 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:32.191936 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.016614 ip-10-0-134-11 kubenswrapper[2571]: I1031 00:14:33.016486 2571 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:14:33.016765 ip-10-0-134-11 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:14:33.033944 ip-10-0-134-11 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:14:33.034107 ip-10-0-134-11 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:14:33.034152 ip-10-0-134-11 systemd[1]: kubelet.service: Consumed 2.562s CPU time. Oct 31 00:14:33.044245 ip-10-0-134-11 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:14:33.118551 ip-10-0-134-11 kubenswrapper[4471]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:14:33.118551 ip-10-0-134-11 kubenswrapper[4471]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:14:33.118551 ip-10-0-134-11 kubenswrapper[4471]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:14:33.118551 ip-10-0-134-11 kubenswrapper[4471]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:14:33.120960 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120864 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.120960 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120957 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.120960 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120963 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.120960 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120968 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120973 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120977 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120981 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120985 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120989 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120993 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.120997 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121001 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121005 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121009 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121014 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121017 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121022 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121026 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121029 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121034 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121046 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121050 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.121244 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121054 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121058 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121062 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121066 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121070 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121075 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121079 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121083 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121087 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121091 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121095 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121100 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121105 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121109 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121113 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121117 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121122 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121127 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121132 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121136 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.122100 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121140 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121145 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121149 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121153 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121157 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121162 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121166 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121171 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121175 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121181 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121185 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121189 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121194 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121198 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121202 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121207 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121211 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121215 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121220 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121224 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.123045 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121228 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121232 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121238 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121243 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121248 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121253 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121257 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121261 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121278 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121283 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121290 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121297 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121302 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121306 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121311 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121315 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121319 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121324 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121328 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.124030 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121332 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121337 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121341 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121345 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121349 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121353 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121358 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121362 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121367 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121371 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121375 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121379 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121384 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121388 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121392 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121399 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121406 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121411 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121415 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.124880 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121420 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121424 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121428 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121435 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121439 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.121443 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122882 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122894 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122899 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122904 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122908 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122913 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122920 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122927 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122933 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122937 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122942 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122948 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122954 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.125785 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122960 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122965 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122970 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122974 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122980 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122984 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122989 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122993 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.122997 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123002 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123006 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123017 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123021 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123025 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123030 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123034 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123039 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123044 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123048 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123052 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.126736 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123056 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123060 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123065 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123069 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123073 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123077 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123082 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123086 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123090 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123095 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123099 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123104 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123108 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123113 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123117 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123121 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123125 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123130 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123134 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123139 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.127604 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123143 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123147 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123153 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123159 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123164 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123168 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123173 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123178 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123183 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123187 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123192 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123196 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123201 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123205 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123209 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123213 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123217 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123221 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123225 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.128145 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123229 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123233 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123237 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123241 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123245 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123249 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123254 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123258 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123263 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123267 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123271 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123276 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123280 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123285 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123289 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123294 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123300 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123305 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123309 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.128806 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123314 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123318 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123322 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123326 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123330 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123335 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123341 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123345 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123350 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123354 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123358 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123362 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123366 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123371 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.123377 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124239 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124249 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124253 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124258 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124262 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.129664 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124266 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124271 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124275 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124280 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124284 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124288 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124293 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124297 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124302 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124310 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124315 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124319 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124324 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124328 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124332 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124337 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124341 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124348 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124353 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.130486 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124357 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124362 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124366 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124370 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124375 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124379 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124383 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124388 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124392 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124396 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124400 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124405 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124409 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124413 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124418 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124422 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124426 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124431 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124435 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124439 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.131209 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124444 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124449 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124455 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124460 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124464 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124469 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124473 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124477 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124481 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124485 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124489 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124495 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124500 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124504 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124510 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124516 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124520 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124541 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124546 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.131744 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124551 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124555 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124560 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124565 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124569 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124573 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124578 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124582 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124587 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124591 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124595 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124600 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124604 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124608 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124613 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124620 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124624 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124629 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124633 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.132208 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124640 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124646 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124651 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124655 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124660 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124665 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124670 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124675 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124679 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124684 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124689 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124693 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124698 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124702 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124706 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124711 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124715 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124719 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124723 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.132686 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124727 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124732 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124736 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124740 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.124746 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124886 4471 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124903 4471 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124913 4471 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124921 4471 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124931 4471 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124936 4471 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124943 4471 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124955 4471 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124961 4471 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124967 4471 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124972 4471 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124977 4471 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124983 4471 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124988 4471 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124993 4471 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.124998 4471 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125003 4471 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125008 4471 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125014 4471 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:14:33.133154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125019 4471 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125024 4471 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125029 4471 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125034 4471 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125042 4471 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125047 4471 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125052 4471 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125057 4471 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125062 4471 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125067 4471 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125072 4471 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125077 4471 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125086 4471 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125093 4471 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125098 4471 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125104 4471 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125108 4471 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125115 4471 flags.go:64] FLAG: --enable-server="true" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125120 4471 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125127 4471 flags.go:64] FLAG: --event-burst="100" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125133 4471 flags.go:64] FLAG: --event-qps="50" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125138 4471 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125143 4471 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125148 4471 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:14:33.133825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125154 4471 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125159 4471 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125165 4471 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125170 4471 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125175 4471 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125181 4471 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125187 4471 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125192 4471 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125196 4471 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125201 4471 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125206 4471 flags.go:64] FLAG: --feature-gates="" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125212 4471 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125217 4471 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125223 4471 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125228 4471 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125233 4471 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125238 4471 flags.go:64] FLAG: --help="false" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125243 4471 flags.go:64] FLAG: --hostname-override="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125248 4471 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125253 4471 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125260 4471 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125266 4471 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125272 4471 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:14:33.134580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125277 4471 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125282 4471 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125289 4471 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125293 4471 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125299 4471 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125304 4471 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125308 4471 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125313 4471 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125318 4471 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125323 4471 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125328 4471 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125333 4471 flags.go:64] FLAG: --lock-file="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125337 4471 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125343 4471 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125348 4471 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125361 4471 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125367 4471 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125372 4471 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125377 4471 flags.go:64] FLAG: --logging-format="text" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125381 4471 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125387 4471 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125391 4471 flags.go:64] FLAG: --manifest-url="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125396 4471 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125403 4471 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125408 4471 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:14:33.135143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125415 4471 flags.go:64] FLAG: --max-pods="110" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125420 4471 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125425 4471 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125430 4471 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125441 4471 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125446 4471 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125451 4471 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125456 4471 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125468 4471 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125474 4471 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125480 4471 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125485 4471 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125490 4471 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125494 4471 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125499 4471 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125504 4471 flags.go:64] FLAG: --port="10250" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125509 4471 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125514 4471 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0ddfff009df4ce2f9" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125519 4471 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125524 4471 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125545 4471 flags.go:64] FLAG: --register-node="true" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125550 4471 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125556 4471 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125562 4471 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125567 4471 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125572 4471 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:14:33.135826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125579 4471 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125583 4471 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125588 4471 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125592 4471 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125597 4471 flags.go:64] FLAG: --runonce="false" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125615 4471 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125622 4471 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125627 4471 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125632 4471 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125637 4471 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125644 4471 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125649 4471 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125654 4471 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125659 4471 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125664 4471 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125670 4471 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125675 4471 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125680 4471 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125685 4471 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125689 4471 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125698 4471 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125704 4471 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125708 4471 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125714 4471 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125719 4471 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:14:33.136447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125724 4471 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125729 4471 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125734 4471 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125738 4471 flags.go:64] FLAG: --v="2" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125745 4471 flags.go:64] FLAG: --version="false" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125753 4471 flags.go:64] FLAG: --vmodule="" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125760 4471 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.125766 4471 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.125991 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.125998 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126003 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126008 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126013 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126017 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126021 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126026 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126030 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126036 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126040 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126045 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126050 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.150383 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126054 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126061 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126066 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126070 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126074 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126079 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126083 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126088 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126092 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126097 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126101 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126105 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126110 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126114 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126119 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126123 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126127 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126133 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126137 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126142 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.151101 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126146 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126150 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126155 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126159 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126163 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126168 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126172 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126177 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126183 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126187 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126191 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126196 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126200 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126207 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126212 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126216 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126221 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126225 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126229 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126234 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.151655 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126238 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126242 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126249 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126256 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126261 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126265 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126270 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126275 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126279 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126283 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126288 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126293 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126297 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126304 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126311 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126315 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126320 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126324 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126329 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.152144 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126333 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126339 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126344 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126348 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126352 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126357 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126362 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126367 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126371 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126375 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126379 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126384 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126388 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126393 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126397 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126402 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126407 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126411 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126415 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126420 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.152682 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126425 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126429 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126434 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126439 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126444 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126448 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126452 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126456 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126460 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126465 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126469 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126474 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126478 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126484 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.126523 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126598 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.153177 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126606 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126611 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126615 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126621 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126626 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126630 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126635 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126639 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126643 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126648 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126652 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126657 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126662 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126667 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126671 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126675 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126680 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126685 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126689 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.153638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126693 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126697 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126703 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126707 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126712 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126716 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126720 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126724 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126728 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126733 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126739 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126742 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126747 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126751 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126754 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126758 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126763 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126768 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126772 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126776 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.154179 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126781 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126785 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126790 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126794 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126798 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126802 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126806 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126811 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126817 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126824 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126829 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126834 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126839 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126844 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126848 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126853 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126857 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126862 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126867 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.154812 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126875 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126880 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126885 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126891 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126896 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126900 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126905 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126909 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126913 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126918 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126922 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126926 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126930 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126935 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126939 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126944 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126948 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126953 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126957 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.155320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126961 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126966 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126970 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126975 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126979 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126983 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126987 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126992 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.126997 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127001 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127006 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127011 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127015 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127019 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127023 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127027 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127034 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127038 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127043 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.155860 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127048 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127052 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127056 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127060 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127067 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127071 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127075 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127079 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.127084 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.132502 4471 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.132540 4471 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.132562 4471 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.132569 4471 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132635 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132641 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132644 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132647 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132651 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132654 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132657 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.156364 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132660 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132662 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132665 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132668 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132671 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132674 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132677 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132680 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132684 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132687 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132690 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132694 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132699 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132703 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132708 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132711 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132714 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132716 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132719 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132721 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.156964 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132724 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132727 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132729 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132732 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132735 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132738 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132741 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132744 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132747 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132750 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132753 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132755 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132758 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132760 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132763 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132766 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132769 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132772 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132777 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132782 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.157484 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132786 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132788 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132791 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132794 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132797 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132799 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132802 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132805 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132807 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132810 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132813 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132815 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132818 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132821 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132823 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132826 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132829 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132832 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132835 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.157998 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132837 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132840 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132842 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132845 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132849 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132853 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132857 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132862 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132865 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132867 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132870 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132872 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132875 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132884 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132887 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132892 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132896 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132899 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132902 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132907 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.158462 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132910 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132913 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132916 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132919 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132922 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132926 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132930 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132934 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132939 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132942 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132945 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132948 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132951 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132953 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132956 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132959 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132962 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132965 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132967 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.158976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.132970 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.133000 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133163 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133169 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133172 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133175 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133178 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133182 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133185 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133188 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133190 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133194 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133196 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133199 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133201 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133204 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.159432 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133207 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133209 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133212 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133215 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133218 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133221 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133223 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133226 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133229 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133233 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133237 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133242 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133246 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133248 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133251 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133253 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133256 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133258 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133261 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133264 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.159962 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133266 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133269 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133273 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133277 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133282 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133285 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133288 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133291 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133293 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133296 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133299 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133303 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133307 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133311 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133315 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133320 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133324 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133327 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133330 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.160465 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133333 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133335 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133338 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133340 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133343 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133345 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133348 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133350 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133353 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133356 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133358 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133361 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133364 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133366 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133369 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133372 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133375 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133377 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133380 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133383 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.160976 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133386 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133388 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133391 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133393 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133396 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133399 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133401 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133404 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133407 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133412 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133416 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133419 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133422 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133424 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133427 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133430 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133432 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133435 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133437 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.161455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133440 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133442 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133445 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133448 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133450 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133453 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133455 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133458 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133461 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133464 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133466 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133469 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133473 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133475 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.133499 4471 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:14:33.161948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133549 4471 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133555 4471 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133558 4471 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133561 4471 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133564 4471 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133566 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133569 4471 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133571 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133574 4471 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133577 4471 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133579 4471 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133582 4471 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133585 4471 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133588 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133590 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133593 4471 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133596 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133599 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133601 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:14:33.162320 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133604 4471 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133606 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133609 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133611 4471 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133614 4471 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133617 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133619 4471 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133624 4471 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133627 4471 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133630 4471 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133632 4471 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133636 4471 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133638 4471 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133642 4471 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133644 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133647 4471 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133649 4471 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133652 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133655 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:14:33.162794 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133657 4471 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133660 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133662 4471 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133665 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133668 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133670 4471 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133673 4471 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133675 4471 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133678 4471 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133681 4471 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133683 4471 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133686 4471 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133688 4471 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133691 4471 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133694 4471 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133696 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133699 4471 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133701 4471 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133704 4471 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133706 4471 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:14:33.163466 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133711 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133713 4471 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133716 4471 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133719 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133722 4471 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133725 4471 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133730 4471 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133733 4471 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133736 4471 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133739 4471 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133742 4471 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133744 4471 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133747 4471 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133749 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133752 4471 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133757 4471 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133765 4471 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133769 4471 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133772 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:14:33.164240 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133775 4471 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:14:33.163504 ip-10-0-134-11 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133778 4471 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133781 4471 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133784 4471 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133787 4471 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133790 4471 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133792 4471 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133795 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133798 4471 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133800 4471 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133803 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133805 4471 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133808 4471 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133812 4471 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133815 4471 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133817 4471 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133820 4471 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133823 4471 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133825 4471 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133829 4471 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133832 4471 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:14:33.165195 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133835 4471 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133838 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133840 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133844 4471 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133847 4471 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133849 4471 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133852 4471 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:33.133854 4471 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.133919 4471 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.134943 4471 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135003 4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135182 4471 server.go:1014] "Starting client certificate rotation" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135325 4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135367 4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:08:52 +0000 UTC" deadline="2028-04-01 11:14:05.139402738 +0000 UTC" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135394 4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12442h59m32.004010517s" Oct 31 00:14:33.166284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.135678 4471 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:14:33.166970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.136762 4471 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:14:33.166970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.138084 4471 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:14:33.166970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.139598 4471 log.go:25] "Validated CRI v1 image API" Oct 31 00:14:33.166970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.140241 4471 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:14:33.166970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.141735 4471 fs.go:135] Filesystem UUIDs: map[3469d953-9944-440d-a4b0-97348d9ac442:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c38f7ac8-59db-45eb-9847-dcf031289c88:/dev/nvme0n1p4] Oct 31 00:14:33.167215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.141756 4471 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fda91ee06c1e2f8018a576b2c3dcf25c740f0f8a686125467a1d5ac4a02bda58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fda91ee06c1e2f8018a576b2c3dcf25c740f0f8a686125467a1d5ac4a02bda58/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~projected/kube-api-access-p5qx8:{mountpoint:/var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~projected/kube-api-access-p5qx8 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~projected/kube-api-access-qww8g:{mountpoint:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~projected/kube-api-access-qww8g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42/volumes/kubernetes.io~projected/kube-api-access-nz4zm:{mountpoint:/var/lib/kubelet/pods/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42/volumes/kubernetes.io~projected/kube-api-access-nz4zm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~projected/kube-api-access-svx5f:{mountpoint:/var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~projected/kube-api-access-svx5f major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37a308b8-65d6-4b6d-a1de-1c7af22ac808/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/37a308b8-65d6-4b6d-a1de-1c7af22ac808/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58180f2e-78da-44ae-8a57-7ad29c0b2e8d/volumes/kubernetes.io~projected/kube-api-access-d6t2r:{mountpoint:/var/lib/kubelet/pods/58180f2e-78da-44ae-8a57-7ad29c0b2e8d/volumes/kubernetes.io~projected/kube-api-access-d6t2r major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ba72de8-84d9-48f4-87b7-49492edd8e48/volumes/kubernetes.io~projected/kube-api-access-xvd46:{mountpoint:/var/lib/kubelet/pods/5ba72de8-84d9-48f4-87b7-49492edd8e48/volumes/kubernetes.io~projected/kube-api-access-xvd46 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b11045e-f44e-4c3c-bce6-18ba33933ab4/volumes/kubernetes.io~projected/kube-api-access-wxtbn:{mountpoint:/var/lib/kubelet/pods/6b11045e-f44e-4c3c-bce6-18ba33933ab4/volumes/kubernetes.io~projected/kube-api-access-wxtbn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f/volumes/kubernetes.io~projected/kube-api-access-www4q:{mountpoint:/var/lib/kubelet/pods/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f/volumes/kubernetes.io~projected/kube-api-access-www4q major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96031c66-2f80-4d17-922c-e168d72ea1a3/volumes/kubernetes.io~projected/kube-api-access-ljsm6:{mountpoint:/var/lib/kubelet/pods/96031c66-2f80-4d17-922c-e168d72ea1a3/volumes/kubernetes.io~projected/kube-api-access-ljsm6 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53fa6db-ce80-464d-a087-dc7f0bad918d/volumes/kubernetes.io~projected/kube-api-access-dc8fm:{mountpoint:/var/lib/kubelet/pods/a53fa6db-ce80-464d-a087-dc7f0bad918d/volumes/kubernetes.io~projected/kube-api-access-dc8fm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~projected/kube-api-access-f4lbf:{mountpoint:/var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~projected/kube-api-access-f4lbf major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~secret/metrics-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5108cf0-9ca6-4eea-b8cf-858b778d5299/volumes/kubernetes.io~projected/kube-api-access-zdt4b:{mountpoint:/var/lib/kubelet/pods/b5108cf0-9ca6-4eea-b8cf-858b778d5299/volumes/kubernetes.io~projected/kube-api-access-zdt4b major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce75cffd-10f6-4f30-a133-72f73169075e/volumes/kubernetes.io~projected/kube-api-access-g28rg:{mountpoint:/var/lib/kubelet/pods/ce75cffd-10f6-4f30-a133-72f73169075e/volumes/kubernetes.io~projected/kube-api-access-g28rg major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~projected/kube-api-access-p24zg:{mountpoint:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~projected/kube-api-access-p24zg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7e9831681e477fb877f4673f2a4410aee554490eefe8e88c8e3d5489a4f3a6d9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9e42e3314e45de695c9f412833d99628aaf73ef0a1b577bb2aa115882cc0d735/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/43776fdd0091135c1a99fbcfa57a9077c04494ffb994913d74a687ffda962df6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2628d5289b78c798c24cc0e1dea77a6266c2f3611b8df0dcd6b5e05418e47506/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/13af6fa63b8e866cd80293815dbc20d8eb3d9ee3dc89230d14f15a543fce6525/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c8a594c8a23021b31a97950d054fc8c3ae5f6433ddd5c1252a63018988591101/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3da072669d5fc907204b55a7fe714dc2836c78cf45cdb924e30eef150868fcd9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7ee0609a1d6709351f0496a398b572eb6dd5616d9d11761e94b01cdaedc1c1ce/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9dd65c4c895c8ef242e0a22d43006fc4ea00a8611dfed1a1a2e833030558a273/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f617c13918c33aeb67fbcf86e655f04722fdd6bd5fc7a5f0b6198ca5838bc6c0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/3a015678f48e17b96f57d6df457fc294fe22f2fcbc315391c2c650a975419eec/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f8d075c855a2523db8caa2485f25c49e7996fdef67267fa2e2a27ce18aa41380/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bf3d4dc68496f0e08b4b72026abf3dd5c74a5042cb690481db9ab5c365277669/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/02d4a4c4ee80307a297853c7964fb8306ee9f990afbf3eac9583b21aa93f7d6d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f35fb717c94869d36f701851fd6b676ca7d06e060246a2e0a024133b3d53d24f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e4e3a313a1632856d16d04344f349cd1f83dbc5a5abf7b0ec9080d116f5a7e12/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/91e01208583f8eb10f8f78d602def6865ae40953b838e39fa39ea1bfc6768331/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e2bd26c65fbaa9af88008a6ab3d0b020a4982e4033508e280af7b8fe692048b0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/964ef7eb9176b9ed6cce4c02ac177a3042805daf0e4ce22e489f8a931f69a0f0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/187b2acf425360926556afd5e5bb59b8573aa924dab37cc2f92c15925eb2c87e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/548748ee74918dc2c83921d40c81a38fb5764962f2c92e3322e197aa5f91be9d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/0ca27845949b7e17d3cb31ca5325f7f27bc083c31918249f8fdbee34eb81c2d7/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/1b14c7f3b5cd25aeac56f10fae3407569b43407eb43dfbf11587def548b8d848/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b5117e3be2e70097ad66b10d8ca17ab79896a74c0c42fa5ca98c9e50ec78bf82/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/a07108d557845227eab543a30ff35ce45d897b515eedb258e6cabdca81327a4e/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/bc5f6658093af193c507ec42acfdf428071c9b9ec4366acaba4fb3129c9802b9/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/4d83c78f510f31df407281afe98cf971731097981ed2296104c09aa3b522ece0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/a48d4ced649bb7a58ec826211dfdca1b4389928a1dc3c5cc1d9aac9e9594a29c/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/fd4f2b9ad164ffdc76cdf881bbc6c3f301a9a35691bfe205449c1f1c39ff1ff5/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/48f227ee49854196d989d625453aa5863f997a51d6ece2c18df226b9e7d4613d/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c4cd107ef7f1b9143f628c4524470272f404bf3414c609f059380121474f1ee5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/650f7efd0a8c0105d2453bdbf9e537f78171a27eb46000377e41b780e0ae07e1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f5affccaa34eef2d442b9e44c1cb3bccc03c6571fb47d69235196719074bd2cd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0d8fbd87a223c15ab418efbb122e8748db07801dc45317130d332bfed7e6515a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0b41be4197af7088e647e79be30b7dc4fe12fb072a4cac1d7b77252f6963ea49/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/4c631695c1a28db1584ed596d9190795553a89de7bf7a34b9d00195c8a8ab0ca/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/f8a7ff6b3784d8d237c31d0426a6aee9e5626a1ec0b3ea30309fad57b0cb2c48/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7dc2e60110048f101dc0d88266be00f822d14938b3c4d307a87bd4d131bf05cf/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d2895077339260a5f45a2e228b550936316fa7f5ad023f5ff2ed2d7f98c98e0d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f4cdc348f7b6b14269032bd3cd5bb6ea2a0eb25038d59cc8ebadf0410319f913/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.148980 4471 manager.go:217] Machine: {Timestamp:2026-10-31 00:14:33.148476178 +0000 UTC m=+0.073745122 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099964 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dfaa2cdb046cfe63b59f4b1adef9a SystemUUID:ec2dfaa2-cdb0-46cf-e63b-59f4b1adef9a BootID:06176286-2e65-4aca-815a-04570f6bfbe3 Filesystems:[{Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42/volumes/kubernetes.io~projected/kube-api-access-nz4zm DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a53fa6db-ce80-464d-a087-dc7f0bad918d/volumes/kubernetes.io~projected/kube-api-access-dc8fm DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce75cffd-10f6-4f30-a133-72f73169075e/volumes/kubernetes.io~projected/kube-api-access-g28rg DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~projected/kube-api-access-f4lbf DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3176d964-e0a4-49f3-b835-6b15b55bb911/volumes/kubernetes.io~projected/kube-api-access-svx5f DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b11045e-f44e-4c3c-bce6-18ba33933ab4/volumes/kubernetes.io~projected/kube-api-access-wxtbn DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ba72de8-84d9-48f4-87b7-49492edd8e48/volumes/kubernetes.io~projected/kube-api-access-xvd46 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~projected/kube-api-access-qww8g DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d46ec9d1d4ca1dbc8607faebe1c46815a1311dbee027cd27f758dabeab870e2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a03cde40e59bf30c43b2838f24c7de816d5c597ea0cdea3187fe7f95d45baec7/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bff9762d47f2fbdb003ceda4126b4b6eb8b3556b965bbb44ca9a66a4d419d724/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cec7cb4d6391ef5b7c9819d87f335d62a6a96ebab8cc4645d2304a958b5d1dc8/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c43f0645bf0ca2c66662d3f04ab7d0d50e3b911f27149e6fcf5f5cde13f98873/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f/volumes/kubernetes.io~projected/kube-api-access-www4q DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb2c36d6d41ddc3cf11fb9b974f370dd0515b2e946c6306e7a37a6db001837a8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78bea2c4c57516ffb15122bcc894ab184ac3849f4da4a45cdf2a3d4642fa0a68/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3be554e7ee8575efc6c0e32c7e5b2ab1743b74c8f8354d0b6944fc71af54971/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96031c66-2f80-4d17-922c-e168d72ea1a3/volumes/kubernetes.io~projected/kube-api-access-ljsm6 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b51aa783695bf3a49d77cbedc1d5ca3fdac2f4aeb826bf372b62a49d7f66caa/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fda91ee06c1e2f8018a576b2c3dcf25c740f0f8a686125467a1d5ac4a02bda58/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a8513b67-029f-448a-ba14-269e06d1f996/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ecc537f1dae8aebdd66e4da97a6829e1318bf6e0f956af554185bd5a137e995/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/16f05275-a5f0-4f35-8f29-fde30538efe4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37a308b8-65d6-4b6d-a1de-1c7af22ac808/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7267046ca7452c49b1bf2f1ca5727b648dca891a3f65810afec58e08fb923829/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5108cf0-9ca6-4eea-b8cf-858b778d5299/volumes/kubernetes.io~projected/kube-api-access-zdt4b DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/58180f2e-78da-44ae-8a57-7ad29c0b2e8d/volumes/kubernetes.io~projected/kube-api-access-d6t2r DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fda5793a-19ef-46b3-8438-51d8f53c46f5/volumes/kubernetes.io~projected/kube-api-access-p24zg DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f6ddd0c-26be-4ef2-a254-791deccb7ae3/volumes/kubernetes.io~projected/kube-api-access-p5qx8 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8d90da693afaa7144f17c52af45fbe6c299f55e1c556931bbe3514949a596c7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:7267046ca7452c4 MacAddress:fe:08:62:3d:57:aa Speed:10000 Mtu:8901} {Name:a03cde40e59bf30 MacAddress:0a:6e:5b:8f:9c:30 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dc:4a:e9:c9 Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:fb:43:fb:99:ce Speed:0 Mtu:8901} {Name:c43f0645bf0ca2c MacAddress:e2:e6:06:3e:72:ef Speed:10000 Mtu:8901} {Name:cec7cb4d6391ef5 MacAddress:be:23:b1:ce:98:3d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dc:4a:e9:c9 Speed:0 Mtu:9001} {Name:fda91ee06c1e2f8 MacAddress:6e:8a:32:de:08:c7 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:6e:16:ab:8c:83:d4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:10:2a:f4:55:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149231 4471 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149277 4471 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149575 4471 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149596 4471 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-11.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149728 4471 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149736 4471 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149746 4471 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149760 4471 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149844 4471 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.149995 4471 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150097 4471 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150131 4471 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150140 4471 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150149 4471 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150160 4471 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150168 4471 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150767 4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.150776 4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.151261 4471 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.158896 4471 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.158907 4471 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159011 4471 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159202 4471 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159229 4471 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159328 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159341 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159347 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159353 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159359 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159365 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159372 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159381 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159391 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159397 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159405 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159415 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159442 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.159450 4471 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.162928 4471 server.go:1263] "Started kubelet" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.162982 4471 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.163120 4471 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.163199 4471 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.163747 4471 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:14:33.167466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.165409 4471 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:14:33.169856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.169659 4471 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:14:33.169856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.169841 4471 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:14:33.170105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.170082 4471 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:14:33.170105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.170104 4471 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:14:33.170271 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.170154 4471 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:08:54 +0000 UTC" deadline="2028-07-14 10:11:39.402937928 +0000 UTC" Oct 31 00:14:33.170271 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.170197 4471 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14937h57m6.232745855s" Oct 31 00:14:33.170271 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.170240 4471 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:14:33.170402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.170282 4471 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:14:33.173567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173484 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42" volumeName="kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm" seLinuxMountContext="" Oct 31 00:14:33.173567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173523 4471 factory.go:55] Registering systemd factory Oct 31 00:14:33.173567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173558 4471 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173523 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" volumeName="kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173640 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f05275-a5f0-4f35-8f29-fde30538efe4" volumeName="kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173664 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" volumeName="kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173677 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" volumeName="kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173700 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ba72de8-84d9-48f4-87b7-49492edd8e48" volumeName="kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173735 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8513b67-029f-448a-ba14-269e06d1f996" volumeName="kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173749 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda5793a-19ef-46b3-8438-51d8f53c46f5" volumeName="kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173780 4471 factory.go:153] Registering CRI-O factory Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173782 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6ddd0c-26be-4ef2-a254-791deccb7ae3" volumeName="kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8" seLinuxMountContext="" Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173803 4471 factory.go:223] Registration of the crio container factory successfully Oct 31 00:14:33.173813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173810 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a308b8-65d6-4b6d-a1de-1c7af22ac808" volumeName="kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173822 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b11045e-f44e-4c3c-bce6-18ba33933ab4" volumeName="kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173848 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8513b67-029f-448a-ba14-269e06d1f996" volumeName="kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173867 4471 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173891 4471 factory.go:103] Registering Raw factory Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173906 4471 manager.go:1199] Started watching for new ooms in manager Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173866 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5108cf0-9ca6-4eea-b8cf-858b778d5299" volumeName="kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173944 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3176d964-e0a4-49f3-b835-6b15b55bb911" volumeName="kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173962 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" volumeName="kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173971 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e5d0b03-f2e3-40d0-93b5-34c19c7be42f" volumeName="kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.173997 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96031c66-2f80-4d17-922c-e168d72ea1a3" volumeName="kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174014 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0f381bf-9848-4fbe-8b12-acd6fb7aedb1" volumeName="kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174031 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda5793a-19ef-46b3-8438-51d8f53c46f5" volumeName="kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174056 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6ddd0c-26be-4ef2-a254-791deccb7ae3" volumeName="kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174072 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f05275-a5f0-4f35-8f29-fde30538efe4" volumeName="kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174084 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b11045e-f44e-4c3c-bce6-18ba33933ab4" volumeName="kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174105 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53fa6db-ce80-464d-a087-dc7f0bad918d" volumeName="kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174116 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53fa6db-ce80-464d-a087-dc7f0bad918d" volumeName="kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174127 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8513b67-029f-448a-ba14-269e06d1f996" volumeName="kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174147 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f6ddd0c-26be-4ef2-a254-791deccb7ae3" volumeName="kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174159 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42" volumeName="kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174170 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42" volumeName="kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174200 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37a308b8-65d6-4b6d-a1de-1c7af22ac808" volumeName="kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca" seLinuxMountContext="" Oct 31 00:14:33.174368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174213 4471 manager.go:319] Starting recovery of all containers Oct 31 00:14:33.175567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.174212 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5108cf0-9ca6-4eea-b8cf-858b778d5299" volumeName="kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca" seLinuxMountContext="" Oct 31 00:14:33.175699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175619 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce75cffd-10f6-4f30-a133-72f73169075e" volumeName="kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg" seLinuxMountContext="" Oct 31 00:14:33.175699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175658 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda5793a-19ef-46b3-8438-51d8f53c46f5" volumeName="kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config" seLinuxMountContext="" Oct 31 00:14:33.175699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175679 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda5793a-19ef-46b3-8438-51d8f53c46f5" volumeName="kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:14:33.175699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175692 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3176d964-e0a4-49f3-b835-6b15b55bb911" volumeName="kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175719 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b11045e-f44e-4c3c-bce6-18ba33933ab4" volumeName="kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175729 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96031c66-2f80-4d17-922c-e168d72ea1a3" volumeName="kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175739 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8513b67-029f-448a-ba14-269e06d1f996" volumeName="kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175758 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda5793a-19ef-46b3-8438-51d8f53c46f5" volumeName="kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175769 4471 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16f05275-a5f0-4f35-8f29-fde30538efe4" volumeName="kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp" seLinuxMountContext="" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175776 4471 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:14:33.175904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.175784 4471 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:14:33.199693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.199659 4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:14:33.200942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.200917 4471 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:14:33.200942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.200941 4471 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:14:33.201060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.200963 4471 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:14:33.201060 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.201003 4471 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:14:33.207748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.207719 4471 generic.go:356] "Generic (PLEG): container finished" podID="cf7e4a92431d081dd5f2682702d947e2" containerID="937289d7e4b9f70925d898fcae0cf1d85a453721731477d8fc73b6eb552e5ff6" exitCode=0 Oct 31 00:14:33.213195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213164 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="d29e1f39e448d75705817aa5af66a327a24375905859ac0d33a1cbc1282797bb" exitCode=0 Oct 31 00:14:33.213195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213191 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="03fbf6855198ce986a6b688c22d5f2cec79cf01da143013dbcc5029df1a69fb8" exitCode=0 Oct 31 00:14:33.213397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213204 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="5db41ee62163e63bd4b1823ca9479b8f2a1208347bf38a089f0d1e1f66c0706f" exitCode=0 Oct 31 00:14:33.213397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213216 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="47b394a64f369a3467d5987bdd997b4bda05afdc8ea6611571e70745cd1e8af6" exitCode=0 Oct 31 00:14:33.213397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213232 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="3788a1ee0ff24d0735a0bc40b8bda7d599e8d0487f01317323a2235fa9eaf449" exitCode=0 Oct 31 00:14:33.213397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.213246 4471 generic.go:356] "Generic (PLEG): container finished" podID="58180f2e-78da-44ae-8a57-7ad29c0b2e8d" containerID="4719f5cd0769eb58618f7930e20e5b69a4be6f286f76ef7841de0a41d41da138" exitCode=0 Oct 31 00:14:33.238983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.238960 4471 manager.go:324] Recovery completed Oct 31 00:14:33.254122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.254098 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.255321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.255302 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.255372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.255335 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.255372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.255345 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.256852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.256835 4471 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:14:33.256852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.256849 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.256944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.256851 4471 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:14:33.256984 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.256946 4471 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:14:33.257129 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257111 4471 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:14:33.257180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257130 4471 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:14:33.257269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257259 4471 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:14:33.257269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257268 4471 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:14:33.257363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257274 4471 policy_none.go:50] "Start" Oct 31 00:14:33.257363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257284 4471 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:14:33.257363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257294 4471 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:14:33.257727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257713 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.257765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257741 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.257765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.257751 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.266312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.266290 4471 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:14:33.266312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.266310 4471 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:14:33.266312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.266317 4471 policy_none.go:44] "Start" Oct 31 00:14:33.270729 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.270679 4471 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:14:33.270908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.270885 4471 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:14:33.274196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274182 4471 manager.go:341] "Starting Device Plugin manager" Oct 31 00:14:33.274274 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.274215 4471 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:14:33.274274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274224 4471 server.go:87] "Starting device plugin registration server" Oct 31 00:14:33.274593 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274579 4471 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:14:33.274662 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274596 4471 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:14:33.274712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274679 4471 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:14:33.274832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274779 4471 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:14:33.274832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274833 4471 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:14:33.274980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274839 4471 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:14:33.274980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.274891 4471 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:14:33.27482266Z","UUID":"468b9bf7-b355-4982-a87a-0ff1bc6fe505","Handler":null,"Name":"","Endpoint":""} Oct 31 00:14:33.276676 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.276136 4471 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:14:33.276676 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.276176 4471 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" Oct 31 00:14:33.276676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.276313 4471 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:14:33.276676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.276336 4471 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:14:33.302122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.302065 4471 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal"] Oct 31 00:14:33.302263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.302222 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.303170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.303152 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.303261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.303181 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.303261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.303191 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.303472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.303460 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.303653 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.303638 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.304276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304261 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.304356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304287 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.304356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304298 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.304356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304265 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.304479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304360 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.304479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304369 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.304554 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.304514 4471 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.304554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304516 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"cf7e4a92431d081dd5f2682702d947e2","Type":"ContainerStarted","Data":"966b6249e63193c15e769de07b574161dd45bcb35edcee126f02ecd1d75140b0"} Oct 31 00:14:33.304645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304631 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.304682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304632 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"682248b34b0664951bb6ec422a0d79db","Type":"ContainerStarted","Data":"2c4f608659111d844cc2b1769cfa73f18d149bc0d743558c6cda5280c3e6567e"} Oct 31 00:14:33.304682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.304674 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" event={"ID":"682248b34b0664951bb6ec422a0d79db","Type":"ContainerStarted","Data":"d2c846814a48d8f52674968613c8f657f7c286c36e65b7606416441b64c3153b"} Oct 31 00:14:33.305295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.305275 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.305386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.305301 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.305386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.305314 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.305495 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.305465 4471 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-11.us-west-2.compute.internal\" not found" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.364616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.364578 4471 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:14:33.364818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.364656 4471 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:14:33.364818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.364814 4471 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:14:33.375988 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.375969 4471 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:14:33.377034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.377018 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:14:33.377118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.377051 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:14:33.377118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.377068 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:14:33.377118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.377104 4471 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.412806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.412782 4471 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:14:33.415059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.415030 4471 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:14:33.465826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.465788 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.465826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.465824 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.465995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.465846 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.470420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.470401 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.514928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.514896 4471 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.515050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.514969 4471 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.515562 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.515546 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeNotReady" Oct 31 00:14:33.515627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.515585 4471 setters.go:546] "Node became not ready" node="ip-10-0-134-11.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:14:33Z","lastTransitionTime":"2026-10-31T00:14:33Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 00:14:33.531516 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.531445 4471 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-11.us-west-2.compute.internal" is owned by different node (oldNodeID="12f8d2a5-55ff-4255-a06d-05741b077195", newNodeID=""), cleaning up... Oct 31 00:14:33.532151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.532121 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:14:33.532236 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.532165 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.532236 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.532179 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566304 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566355 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566382 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566432 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/682248b34b0664951bb6ec422a0d79db-config\") pod \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"682248b34b0664951bb6ec422a0d79db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566434 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.566476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.566436 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf7e4a92431d081dd5f2682702d947e2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" (UID: \"cf7e4a92431d081dd5f2682702d947e2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:33.631025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:33.630989 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:14:33.631164 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:33.631049 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:34.150974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.150943 4471 apiserver.go:52] "Watching apiserver" Oct 31 00:14:34.242263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.242228 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:34.242423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.242350 4471 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:34.322295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.322257 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:14:34.322459 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.322306 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:34.322459 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.322255 4471 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:14:34.322459 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.322439 4471 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal" Oct 31 00:14:34.322665 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.322482 4471 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:14:34.322977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.322959 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-rzsvb","kube-system/kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm","openshift-service-ca/service-ca-5fffc97bdf-42jns","openshift-multus/multus-additional-cni-plugins-ncrxv","openshift-image-registry/node-ca-s6mps","openshift-ovn-kubernetes/ovnkube-node-8x6zh","kube-system/konnectivity-agent-4rxkn","openshift-dns/node-resolver-82m77","openshift-multus/multus-nz77q","openshift-cluster-node-tuning-operator/tuned-qh76s","openshift-ingress-canary/ingress-canary-wj8wj","openshift-network-operator/iptables-alerter-4nxqs","kube-system/global-pull-secret-syncer-t57nz","openshift-dns/dns-default-5c5ss","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-11.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-8nzvb","openshift-insights/insights-runtime-extractor-t25p7"] Oct 31 00:14:34.324126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.324105 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.332219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.332169 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.371333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371292 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qx8\" (UniqueName: \"kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371343 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371371 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371394 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371445 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371476 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371494 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.371582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371522 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.371897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371614 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvd46\" (UniqueName: \"kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.371897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371667 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8fm\" (UniqueName: \"kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.371897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371747 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.371897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.371812 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372008 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372043 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372071 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372086 4471 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:14:34.372205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372121 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdt4b\" (UniqueName: \"kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.372272 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372254 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-www4q\" (UniqueName: \"kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.372334 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372292 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372334 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372321 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372359 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.372427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372378 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.372427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372395 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372427 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372460 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fa6db-ce80-464d-a087-dc7f0bad918d-tmp-dir\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372461 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372490 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372519 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372566 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.372603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372595 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372623 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372646 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372664 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6t2r\" (UniqueName: \"kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372688 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372712 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372738 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372766 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372807 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372849 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372873 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.372895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372889 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qww8g\" (UniqueName: \"kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372917 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx5f\" (UniqueName: \"kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372938 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-tmp\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372952 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.372977 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373000 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373030 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373052 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373072 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373108 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24zg\" (UniqueName: \"kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373150 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373234 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373254 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373270 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373285 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373299 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373341 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373368 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-tuned\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373376 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373402 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:34.373427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373424 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373469 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.373973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373487 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373502 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.373973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373516 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.373973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.373542 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.376377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.376353 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:34.380523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.380506 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:14:34.381197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.381182 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ds5xr\"" Oct 31 00:14:34.382184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382164 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:14:34.382264 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382250 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.382331 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382314 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovn-node-metrics-cert\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382450 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382605 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382620 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382629 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382641 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7hvcc\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382674 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382682 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382696 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bvv2r\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382702 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382719 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382722 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382746 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382771 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382774 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382786 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.382814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382800 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382908 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h9nxs\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382923 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382951 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382953 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdpqj\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382967 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v2pb4\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382952 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.382980 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383058 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383066 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383102 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zk5sj\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383376 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383398 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:14:34.383433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383439 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383450 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bcxbn\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383502 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383569 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lg2hb\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383621 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfr46\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383636 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-env-overrides\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383572 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-plmbm\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383678 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383697 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383707 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383817 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383820 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383833 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5jgbw\"" Oct 31 00:14:34.383975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.383922 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.392598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.392581 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.392685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.392665 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-config\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.392768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.392749 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.393777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.393762 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3176d964-e0a4-49f3-b835-6b15b55bb911-cert\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:34.393865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.393850 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fda5793a-19ef-46b3-8438-51d8f53c46f5-ovnkube-script-lib\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.394302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.394280 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5108cf0-9ca6-4eea-b8cf-858b778d5299-serviceca\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.423505 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.423435 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.427886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.427855 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:14:34.427886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.427884 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.428036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.427889 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:14:34.428036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.427908 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 00:14:34.428036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.427919 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.429510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.429497 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.430322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.430306 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:14:34.430388 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.430345 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qszwm\"" Oct 31 00:14:34.431010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.430973 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r9nst\"" Oct 31 00:14:34.431071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.430989 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 00:14:34.431121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431090 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:14:34.431121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431097 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-74dxc\"" Oct 31 00:14:34.431230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431216 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:14:34.431230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431219 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:14:34.431327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431249 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hcz6l\"" Oct 31 00:14:34.431761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-www4q\" (UniqueName: \"kubernetes.io/projected/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-kube-api-access-www4q\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431918 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-key\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.431931 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvd46\" (UniqueName: \"kubernetes.io/projected/5ba72de8-84d9-48f4-87b7-49492edd8e48-kube-api-access-xvd46\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432022 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdt4b\" (UniqueName: \"kubernetes.io/projected/b5108cf0-9ca6-4eea-b8cf-858b778d5299-kube-api-access-zdt4b\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432079 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qx8\" (UniqueName: \"kubernetes.io/projected/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-kube-api-access-p5qx8\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432237 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6t2r\" (UniqueName: \"kubernetes.io/projected/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-kube-api-access-d6t2r\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8fm\" (UniqueName: \"kubernetes.io/projected/a53fa6db-ce80-464d-a087-dc7f0bad918d-kube-api-access-dc8fm\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.432426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432394 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qww8g\" (UniqueName: \"kubernetes.io/projected/16f05275-a5f0-4f35-8f29-fde30538efe4-kube-api-access-qww8g\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.432704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432661 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24zg\" (UniqueName: \"kubernetes.io/projected/fda5793a-19ef-46b3-8438-51d8f53c46f5-kube-api-access-p24zg\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.432704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.432682 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx5f\" (UniqueName: \"kubernetes.io/projected/3176d964-e0a4-49f3-b835-6b15b55bb911-kube-api-access-svx5f\") pod \"ingress-canary-wj8wj\" (UID: \"3176d964-e0a4-49f3-b835-6b15b55bb911\") " pod="openshift-ingress-canary/ingress-canary-wj8wj" Oct 31 00:14:34.434454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.434441 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f6ddd0c-26be-4ef2-a254-791deccb7ae3-signing-cabundle\") pod \"service-ca-5fffc97bdf-42jns\" (UID: \"0f6ddd0c-26be-4ef2-a254-791deccb7ae3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-42jns" Oct 31 00:14:34.474146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474115 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.474146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474152 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.474341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4zm\" (UniqueName: \"kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.474341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474188 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:34.474341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474248 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.474341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474306 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.474498 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474368 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8513b67-029f-448a-ba14-269e06d1f996-tmp-dir\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.474498 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474476 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.474624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474509 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.474624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474521 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-sys\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.474624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474588 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474641 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474678 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474683 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474699 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37a308b8-65d6-4b6d-a1de-1c7af22ac808-konnectivity-ca\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474720 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-etc-selinux\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.474762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474746 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474765 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-kubernetes\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474775 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474791 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-slash\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474806 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474811 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-netd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474736 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-socket-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474844 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474865 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474870 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-run\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474884 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474906 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474911 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-system-cni-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474925 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-etc-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474935 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474948 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474951 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474967 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474978 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5108cf0-9ca6-4eea-b8cf-858b778d5299-host\") pod \"node-ca-s6mps\" (UID: \"b5108cf0-9ca6-4eea-b8cf-858b778d5299\") " pod="openshift-image-registry/node-ca-s6mps" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474966 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-var-lib-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.474992 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475014 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-host\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475019 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.475048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475031 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-ovn\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.475070 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.475145 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:34.975129677 +0000 UTC m=+1.900398648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : secret "metrics-daemon-secret" not found Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475160 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtbn\" (UniqueName: \"kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475178 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475197 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475211 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475228 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475241 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475252 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-cnibin\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475255 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475293 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475297 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-systemd\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475323 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a53fa6db-ce80-464d-a087-dc7f0bad918d-hosts-file\") pod \"node-resolver-82m77\" (UID: \"a53fa6db-ce80-464d-a087-dc7f0bad918d\") " pod="openshift-dns/node-resolver-82m77" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475330 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475355 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58180f2e-78da-44ae-8a57-7ad29c0b2e8d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncrxv\" (UID: \"58180f2e-78da-44ae-8a57-7ad29c0b2e8d\") " pod="openshift-multus/multus-additional-cni-plugins-ncrxv" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475357 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475381 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475392 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-node-log\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475405 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475420 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-log-socket\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475431 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475455 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475462 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-registration-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475483 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475497 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-systemd\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475507 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475509 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-var-lib-kubelet\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475548 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cni-binary-copy\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475569 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475640 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475659 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475682 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475706 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475729 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475750 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475774 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475761 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-systemd-units\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475789 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-run-openvswitch\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475795 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8513b67-029f-448a-ba14-269e06d1f996-config-volume\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475729 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysconfig\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475805 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-run-netns\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475803 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475840 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475845 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-cni-bin\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475869 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475888 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475904 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475913 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-original-pull-secret\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475962 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475976 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-conf\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.475990 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476003 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-sys-fs\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476008 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-modprobe-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476033 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476062 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476087 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.476080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476107 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37a308b8-65d6-4b6d-a1de-1c7af22ac808-agent-certs\") pod \"konnectivity-agent-4rxkn\" (UID: \"37a308b8-65d6-4b6d-a1de-1c7af22ac808\") " pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476140 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476160 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-daemon-config\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476171 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-etc-sysctl-d\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476196 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsm6\" (UniqueName: \"kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476293 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476326 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476337 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-kubelet\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476349 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476377 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96031c66-2f80-4d17-922c-e168d72ea1a3-iptables-alerter-script\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476390 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lbf\" (UniqueName: \"kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476542 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476580 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476606 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476606 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-system-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476582 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8513b67-029f-448a-ba14-269e06d1f996-metrics-tls\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476629 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476650 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-data-volume\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476670 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476671 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ba72de8-84d9-48f4-87b7-49492edd8e48-device-dir\") pod \"aws-ebs-csi-driver-node-nrzfm\" (UID: \"5ba72de8-84d9-48f4-87b7-49492edd8e48\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nrzfm" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476670 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fda5793a-19ef-46b3-8438-51d8f53c46f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8x6zh\" (UID: \"fda5793a-19ef-46b3-8438-51d8f53c46f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:34.477906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.476728 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16f05275-a5f0-4f35-8f29-fde30538efe4-lib-modules\") pod \"tuned-qh76s\" (UID: \"16f05275-a5f0-4f35-8f29-fde30538efe4\") " pod="openshift-cluster-node-tuning-operator/tuned-qh76s" Oct 31 00:14:34.538107 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.538060 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4zm\" (UniqueName: \"kubernetes.io/projected/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-kube-api-access-nz4zm\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.539063 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.539039 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtbn\" (UniqueName: \"kubernetes.io/projected/6b11045e-f44e-4c3c-bce6-18ba33933ab4-kube-api-access-wxtbn\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.539180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.539131 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lbf\" (UniqueName: \"kubernetes.io/projected/a8513b67-029f-448a-ba14-269e06d1f996-kube-api-access-f4lbf\") pod \"dns-default-5c5ss\" (UID: \"a8513b67-029f-448a-ba14-269e06d1f996\") " pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:34.539266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.539247 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsm6\" (UniqueName: \"kubernetes.io/projected/96031c66-2f80-4d17-922c-e168d72ea1a3-kube-api-access-ljsm6\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.539464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.539442 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28rg\" (UniqueName: \"kubernetes.io/projected/ce75cffd-10f6-4f30-a133-72f73169075e-kube-api-access-g28rg\") pod \"network-check-target-rzsvb\" (UID: \"ce75cffd-10f6-4f30-a133-72f73169075e\") " pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:34.577934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.577906 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.577934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.577935 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.577953 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.577970 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.577991 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578031 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578046 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-cnibin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578054 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-cni-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578061 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578088 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-multus\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578094 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-k8s-cni-cncf-io\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578104 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-netns\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578102 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578132 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-socket-dir-parent\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578134 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-hostroot\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578136 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578158 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-multus-conf-dir\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578161 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-cni-bin\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578173 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578200 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578226 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-var-lib-kubelet\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578255 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578265 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-etc-kubernetes\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578279 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578303 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-kubelet-config\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.578357 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578360 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f381bf-9848-4fbe-8b12-acd6fb7aedb1-dbus\") pod \"global-pull-secret-syncer-t57nz\" (UID: \"a0f381bf-9848-4fbe-8b12-acd6fb7aedb1\") " pod="kube-system/global-pull-secret-syncer-t57nz" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.578417 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:35.078400915 +0000 UTC m=+2.003669847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578432 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578460 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578483 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578484 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b11045e-f44e-4c3c-bce6-18ba33933ab4-host-run-multus-certs\") pod \"multus-nz77q\" (UID: \"6b11045e-f44e-4c3c-bce6-18ba33933ab4\") " pod="openshift-multus/multus-nz77q" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578506 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96031c66-2f80-4d17-922c-e168d72ea1a3-host-slash\") pod \"iptables-alerter-4nxqs\" (UID: \"96031c66-2f80-4d17-922c-e168d72ea1a3\") " pod="openshift-network-operator/iptables-alerter-4nxqs" Oct 31 00:14:34.578933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.578562 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-crio-socket\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:34.635212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.635192 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:14:34.981324 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:34.981281 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:34.981498 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.981420 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:14:34.981498 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:34.981480 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:35.981465497 +0000 UTC m=+2.906734433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : secret "metrics-daemon-secret" not found Oct 31 00:14:35.082186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.082149 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:35.082359 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:35.082293 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:35.082396 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:35.082359 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:36.082342396 +0000 UTC m=+3.007611335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:35.368220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.368143 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:35.392991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.392960 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.595214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.595179 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.746089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.746057 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.887798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.887763 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.989696 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:35.989656 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:35.989904 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:35.989800 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:14:35.989904 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:35.989881 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:37.989862504 +0000 UTC m=+4.915131460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : secret "metrics-daemon-secret" not found Oct 31 00:14:36.033110 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.033010 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.050174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.050147 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:14:36.090951 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.090919 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:36.091140 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:36.091028 4471 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 00:14:36.091140 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:36.091088 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls podName:30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42 nodeName:}" failed. No retries permitted until 2026-10-31 00:14:38.091071847 +0000 UTC m=+5.016340780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t25p7" (UID: "30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42") : secret "insights-runtime-extractor-tls" not found Oct 31 00:14:36.175737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.175698 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.318274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.318192 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.455771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.455732 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:36.456197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.456029 4471 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:14:36.466245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:36.466219 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:37.397245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:37.397199 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:37.598100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:37.598068 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:37.794521 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:37.794446 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:38.006972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.006948 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:38.007072 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:38.007061 4471 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 00:14:38.007115 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:14:38.007108 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs podName:6e5d0b03-f2e3-40d0-93b5-34c19c7be42f nodeName:}" failed. No retries permitted until 2026-10-31 00:14:42.007094617 +0000 UTC m=+8.932363563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs") pod "network-metrics-daemon-8nzvb" (UID: "6e5d0b03-f2e3-40d0-93b5-34c19c7be42f") : secret "metrics-daemon-secret" not found Oct 31 00:14:38.094306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.094194 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:38.107500 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.107466 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:38.107864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.107847 4471 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:14:38.109872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.109856 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t25p7\" (UID: \"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42\") " pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:38.233586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.233554 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t25p7" Oct 31 00:14:38.244958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.244932 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:38.256832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.256802 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c5ss" event={"ID":"a8513b67-029f-448a-ba14-269e06d1f996","Type":"ContainerStarted","Data":"6aa7af546eb364d848027f7daf8887845929e738af194cf5b5495f6d30f19366"} Oct 31 00:14:38.256978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.256869 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5c5ss" event={"ID":"a8513b67-029f-448a-ba14-269e06d1f996","Type":"ContainerStarted","Data":"7bf82f4c469233417340d7ed640480aa2645eb6a180655dc903b74d0c1125be5"} Oct 31 00:14:38.257076 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.257060 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:38.598425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:38.598398 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t25p7"] Oct 31 00:14:38.598851 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:38.598484 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ef8ee6_cd87_4ca1_8e6a_9759f9e7ad42.slice/crio-7e55cfbde6c20f1ff789857141be2f9f686a82353b10173021e7965b581d57b7 WatchSource:0}: Error finding container 7e55cfbde6c20f1ff789857141be2f9f686a82353b10173021e7965b581d57b7: Status 404 returned error can't find the container with id 7e55cfbde6c20f1ff789857141be2f9f686a82353b10173021e7965b581d57b7 Oct 31 00:14:39.262170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.262138 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t25p7" event={"ID":"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42","Type":"ContainerStarted","Data":"bf635ee627a063ef7cf7d9547b0deb3a128f580545aa983bdd5906c2eb204c5e"} Oct 31 00:14:39.262349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.262194 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t25p7" event={"ID":"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42","Type":"ContainerStarted","Data":"7e55cfbde6c20f1ff789857141be2f9f686a82353b10173021e7965b581d57b7"} Oct 31 00:14:39.395953 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.395920 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.612991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.612909 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.795659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.795623 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.999231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:39.999196 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.192495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:40.192473 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.267562 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:40.267471 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t25p7" event={"ID":"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42","Type":"ContainerStarted","Data":"9c1ebdefb01c9819b61079456ee28b31504bb07ccbfedcf0f93d5bad30ce4bff"} Oct 31 00:14:41.651994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:41.651961 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8x6zh" Oct 31 00:14:42.033662 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:42.033572 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:42.036347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:42.036321 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d0b03-f2e3-40d0-93b5-34c19c7be42f-metrics-certs\") pod \"network-metrics-daemon-8nzvb\" (UID: \"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f\") " pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:42.125397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:42.125360 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nzvb" Oct 31 00:14:42.494443 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:42.494409 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5d0b03_f2e3_40d0_93b5_34c19c7be42f.slice/crio-bfe22a2fcb54c28fedf9e9277bfce8c47d86ea7907982fbd5046725e9313cb4b WatchSource:0}: Error finding container bfe22a2fcb54c28fedf9e9277bfce8c47d86ea7907982fbd5046725e9313cb4b: Status 404 returned error can't find the container with id bfe22a2fcb54c28fedf9e9277bfce8c47d86ea7907982fbd5046725e9313cb4b Oct 31 00:14:42.494716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:42.494693 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nzvb"] Oct 31 00:14:43.281563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:43.278899 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nzvb" event={"ID":"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f","Type":"ContainerStarted","Data":"bfe22a2fcb54c28fedf9e9277bfce8c47d86ea7907982fbd5046725e9313cb4b"} Oct 31 00:14:43.282035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:43.281668 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t25p7" event={"ID":"30ef8ee6-cd87-4ca1-8e6a-9759f9e7ad42","Type":"ContainerStarted","Data":"28e111d18333f7394243b4b1da59bfe17f1d8540e701b2558af61c12f594d50f"} Oct 31 00:14:43.622725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:43.622653 4471 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-11.us-west-2.compute.internal" event="NodeReady" Oct 31 00:14:44.286230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:44.286198 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nzvb" event={"ID":"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f","Type":"ContainerStarted","Data":"f983985123a09522179600f0bc9ec80df27540a03aa0d1b0937a36f341455cfb"} Oct 31 00:14:44.745207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:44.745170 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:44.898720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:44.898673 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:45.199952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:45.199887 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:45.291511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:45.291479 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nzvb" event={"ID":"6e5d0b03-f2e3-40d0-93b5-34c19c7be42f","Type":"ContainerStarted","Data":"144d91c083a28fe943e9cbf3ad50c08b7300a6985bea46be7fcb614b9b89550b"} Oct 31 00:14:45.348994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:45.348913 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:48.264069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:48.264037 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5c5ss" Oct 31 00:14:51.791596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:51.791553 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:51.935277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:51.935230 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.080701 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:52.080631 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.539943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:52.539916 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4rxkn" Oct 31 00:14:52.838048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:52.837970 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.916064 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:52.916036 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp"] Oct 31 00:14:52.918485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:52.918466 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:53.012609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.012575 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee61e853-22b2-487f-96a2-8ce7aa360d06-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-srvsp\" (UID: \"ee61e853-22b2-487f-96a2-8ce7aa360d06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:53.022867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.022829 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 00:14:53.023012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.022872 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-m8x85\"" Oct 31 00:14:53.023382 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.023364 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp"] Oct 31 00:14:53.051127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.051095 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.113378 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.113304 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee61e853-22b2-487f-96a2-8ce7aa360d06-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-srvsp\" (UID: \"ee61e853-22b2-487f-96a2-8ce7aa360d06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:53.115711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.115692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee61e853-22b2-487f-96a2-8ce7aa360d06-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-srvsp\" (UID: \"ee61e853-22b2-487f-96a2-8ce7aa360d06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:53.198795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.198761 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.229863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.229825 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:53.350426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.350370 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.593367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:53.593340 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp"] Oct 31 00:14:53.594075 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:53.594045 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee61e853_22b2_487f_96a2_8ce7aa360d06.slice/crio-1e2481aed05412d69f804f074c13502bc1b581a2b3e9689aa30b4af89c6878d1 WatchSource:0}: Error finding container 1e2481aed05412d69f804f074c13502bc1b581a2b3e9689aa30b4af89c6878d1: Status 404 returned error can't find the container with id 1e2481aed05412d69f804f074c13502bc1b581a2b3e9689aa30b4af89c6878d1 Oct 31 00:14:54.323165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:54.323126 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" event={"ID":"ee61e853-22b2-487f-96a2-8ce7aa360d06","Type":"ContainerStarted","Data":"1e2481aed05412d69f804f074c13502bc1b581a2b3e9689aa30b4af89c6878d1"} Oct 31 00:14:54.468814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:54.468779 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:54.616743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:54.616650 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:54.764694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:54.764655 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:56.330674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:56.330636 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" event={"ID":"ee61e853-22b2-487f-96a2-8ce7aa360d06","Type":"ContainerStarted","Data":"fa12de796b6df74f0ecf54d756bc56f69ade0aa79a2c25387489a296767eeff1"} Oct 31 00:14:56.331177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:56.330843 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:56.335307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:56.335285 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" Oct 31 00:14:56.445678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:56.445622 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-srvsp" podStartSLOduration=2.501137728 podStartE2EDuration="4.445608249s" totalImagesPullingTime="1.944470521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:56.445252071 +0000 UTC m=+23.370521024" watchObservedRunningTime="2026-10-31 00:14:56.445608249 +0000 UTC m=+23.370877203" Oct 31 00:14:57.106293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.106260 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.253353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.253307 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.416466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.416417 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.496655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.496624 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rj944"] Oct 31 00:14:57.498776 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.498759 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.537762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.537724 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.537928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.537769 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.537928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.537799 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qd6\" (UniqueName: \"kubernetes.io/projected/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-kube-api-access-56qd6\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.537928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.537892 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.564174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.564138 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.600901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.600870 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 00:14:57.601120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.600916 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:14:57.601120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.601018 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:14:57.601120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.601056 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 00:14:57.601120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.601092 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ptxp2\"" Oct 31 00:14:57.601120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.601112 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:14:57.638784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.638753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.638916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.638791 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.638916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.638832 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qd6\" (UniqueName: \"kubernetes.io/projected/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-kube-api-access-56qd6\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.638916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.638862 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.639611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.639583 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.641221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.641200 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.641297 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.641224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.651369 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.651347 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rj944"] Oct 31 00:14:57.700081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.700015 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qd6\" (UniqueName: \"kubernetes.io/projected/4fd3ce17-9e54-4115-9e24-82ea6ac4551c-kube-api-access-56qd6\") pod \"prometheus-operator-9cd89b74c-rj944\" (UID: \"4fd3ce17-9e54-4115-9e24-82ea6ac4551c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.810143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.810089 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" Oct 31 00:14:57.815751 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.815710 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:57.965803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:57.965709 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.113421 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.113391 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.171340 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:14:58.171309 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd3ce17_9e54_4115_9e24_82ea6ac4551c.slice/crio-4a426835cbd0681a6d804a657f34b523726d0d2eff72eed7256ea2237e71ac9b WatchSource:0}: Error finding container 4a426835cbd0681a6d804a657f34b523726d0d2eff72eed7256ea2237e71ac9b: Status 404 returned error can't find the container with id 4a426835cbd0681a6d804a657f34b523726d0d2eff72eed7256ea2237e71ac9b Oct 31 00:14:58.171649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.171631 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rj944"] Oct 31 00:14:58.257209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.257140 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.338440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.338409 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" event={"ID":"4fd3ce17-9e54-4115-9e24-82ea6ac4551c","Type":"ContainerStarted","Data":"4a426835cbd0681a6d804a657f34b523726d0d2eff72eed7256ea2237e71ac9b"} Oct 31 00:14:58.402317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.402279 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.548321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.548250 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.826689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.826606 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:58.970942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:14:58.970893 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.640384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:00.640360 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:00.794367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:00.794287 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:01.349599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:01.349561 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" event={"ID":"4fd3ce17-9e54-4115-9e24-82ea6ac4551c","Type":"ContainerStarted","Data":"68ba7228104711ff2d27b2ab6e560691a55d4548d9c075234f5b17fe3f44208f"} Oct 31 00:15:01.349774 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:01.349618 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" event={"ID":"4fd3ce17-9e54-4115-9e24-82ea6ac4551c","Type":"ContainerStarted","Data":"cff38e2be3f6c8752960928a1b715b4481fa5765fc07fd097af44ae12c63082f"} Oct 31 00:15:01.466506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:01.466458 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rj944" podStartSLOduration=2.178811276 podStartE2EDuration="4.466443805s" totalImagesPullingTime="2.287632529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:01.465918194 +0000 UTC m=+28.391187148" watchObservedRunningTime="2026-10-31 00:15:01.466443805 +0000 UTC m=+28.391712759" Oct 31 00:15:01.996106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:01.996076 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:02.147171 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:02.147134 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:02.990910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:02.990878 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:03.495402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:03.495369 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:03.691209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:03.691173 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:03.919120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:03.919085 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.070028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.069993 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.252384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.252265 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.262610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.262586 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k9r7z"] Oct 31 00:15:04.280050 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:15:04.280017 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:15:04.292668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.292643 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.347871 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.347840 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:15:04.348102 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.348087 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:15:04.348156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.348103 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ks82c\"" Oct 31 00:15:04.350405 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.350391 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:15:04.385754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385716 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hrx\" (UniqueName: \"kubernetes.io/projected/48004d0a-ef43-4245-8788-a0f823e3c8da-kube-api-access-v8hrx\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.385754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385753 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-tls\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.385972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385773 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-accelerators-collector-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.385972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385804 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-root\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.385972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-metrics-client-ca\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.385972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385958 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-textfile\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.386129 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.385980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-wtmp\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.386129 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.386028 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-sys\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.386129 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.386079 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.460437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.460400 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.487314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487279 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-root\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487320 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-metrics-client-ca\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487341 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-textfile\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487362 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-wtmp\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487389 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-sys\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487394 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-root\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487410 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487462 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hrx\" (UniqueName: \"kubernetes.io/projected/48004d0a-ef43-4245-8788-a0f823e3c8da-kube-api-access-v8hrx\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487490 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-tls\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487510 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-sys\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487513 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-accelerators-collector-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487587 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-wtmp\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.487780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.487690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-textfile\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.488082 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.488054 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-accelerators-collector-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.488384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.488364 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48004d0a-ef43-4245-8788-a0f823e3c8da-metrics-client-ca\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.489802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.489777 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.490080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.490057 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48004d0a-ef43-4245-8788-a0f823e3c8da-node-exporter-tls\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.548124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.548045 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hrx\" (UniqueName: \"kubernetes.io/projected/48004d0a-ef43-4245-8788-a0f823e3c8da-kube-api-access-v8hrx\") pod \"node-exporter-k9r7z\" (UID: \"48004d0a-ef43-4245-8788-a0f823e3c8da\") " pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.607294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.607264 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:04.641385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.641354 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k9r7z" Oct 31 00:15:04.681993 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:15:04.681953 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48004d0a_ef43_4245_8788_a0f823e3c8da.slice/crio-4afceb1b2f0351667f9d94a93fff5a8557727b7994735c8cb79217b5cec3ed40 WatchSource:0}: Error finding container 4afceb1b2f0351667f9d94a93fff5a8557727b7994735c8cb79217b5cec3ed40: Status 404 returned error can't find the container with id 4afceb1b2f0351667f9d94a93fff5a8557727b7994735c8cb79217b5cec3ed40 Oct 31 00:15:04.839119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:04.839048 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.364117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:05.364077 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k9r7z" event={"ID":"48004d0a-ef43-4245-8788-a0f823e3c8da","Type":"ContainerStarted","Data":"4afceb1b2f0351667f9d94a93fff5a8557727b7994735c8cb79217b5cec3ed40"} Oct 31 00:15:05.629419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:05.629341 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:05.906509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:05.906469 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:06.053993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:06.053968 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rzsvb" Oct 31 00:15:06.479460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:06.479435 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:06.912472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:06.912433 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:07.005251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.005218 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6fb76784f8-nc45q"] Oct 31 00:15:07.007903 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.007885 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.025329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.025299 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fb76784f8-nc45q"] Oct 31 00:15:07.080239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.080206 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:07.082461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082429 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:15:07.082603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082492 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:15:07.082603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082495 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:15:07.082603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082570 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:15:07.082724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082624 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gqv5f\"" Oct 31 00:15:07.082724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082663 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:15:07.082724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.082682 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8i2q115mts7m2\"" Oct 31 00:15:07.107071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107038 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107072 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107109 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-grpc-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107184 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwdh\" (UniqueName: \"kubernetes.io/projected/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-kube-api-access-vfwdh\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107207 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-tmp\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107228 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107251 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-metrics-client-ca\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107334 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.107435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.107353 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208097 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208138 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208153 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208284 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-grpc-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208320 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwdh\" (UniqueName: \"kubernetes.io/projected/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-kube-api-access-vfwdh\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208342 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-tmp\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208372 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208399 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-metrics-client-ca\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208444 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.208923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.208866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-tmp\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.209358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.209300 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-metrics-client-ca\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.210985 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.211016 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.211197 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.211231 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.211301 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-grpc-tls\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.211387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.211363 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.269746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.269717 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwdh\" (UniqueName: \"kubernetes.io/projected/7afe3f17-55c3-4edd-bfce-ddf42f5230f5-kube-api-access-vfwdh\") pod \"thanos-querier-6fb76784f8-nc45q\" (UID: \"7afe3f17-55c3-4edd-bfce-ddf42f5230f5\") " pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.326422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.326390 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:07.375248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.374021 4471 generic.go:356] "Generic (PLEG): container finished" podID="48004d0a-ef43-4245-8788-a0f823e3c8da" containerID="ee4c1c779492e65480013346534db092c4dc9867712d3db67ba157844f61ea99" exitCode=0 Oct 31 00:15:07.375248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.374130 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k9r7z" event={"ID":"48004d0a-ef43-4245-8788-a0f823e3c8da","Type":"ContainerDied","Data":"ee4c1c779492e65480013346534db092c4dc9867712d3db67ba157844f61ea99"} Oct 31 00:15:07.683225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.683183 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:07.747993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.747962 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fb76784f8-nc45q"] Oct 31 00:15:07.747993 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:15:07.747973 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afe3f17_55c3_4edd_bfce_ddf42f5230f5.slice/crio-d03d9aeb26493b1c03a9eaf04a1f647ca78e2dce164d4a6cbf6a1858afcc9949 WatchSource:0}: Error finding container d03d9aeb26493b1c03a9eaf04a1f647ca78e2dce164d4a6cbf6a1858afcc9949: Status 404 returned error can't find the container with id d03d9aeb26493b1c03a9eaf04a1f647ca78e2dce164d4a6cbf6a1858afcc9949 Oct 31 00:15:07.939243 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:07.939212 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.132683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.132579 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.379515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.379479 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"d03d9aeb26493b1c03a9eaf04a1f647ca78e2dce164d4a6cbf6a1858afcc9949"} Oct 31 00:15:08.381843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.381814 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k9r7z" event={"ID":"48004d0a-ef43-4245-8788-a0f823e3c8da","Type":"ContainerStarted","Data":"ecf0e028b023b1c57fa60b235465448f31a7cbcb48afce72aec5a020c5081782"} Oct 31 00:15:08.381983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.381860 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k9r7z" event={"ID":"48004d0a-ef43-4245-8788-a0f823e3c8da","Type":"ContainerStarted","Data":"49a8e111773188a43fbec0bf394935f895160b99dca67f211cb1c026584b898d"} Oct 31 00:15:08.451317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.451290 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.510959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.510907 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k9r7z" podStartSLOduration=2.6855722269999998 podStartE2EDuration="4.510892469s" totalImagesPullingTime="1.75710416s" totalInitContainerRuntime="68.216082ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:08.51057152 +0000 UTC m=+35.435840476" watchObservedRunningTime="2026-10-31 00:15:08.510892469 +0000 UTC m=+35.436161423" Oct 31 00:15:08.641548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.641506 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.842048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.841976 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:08.984278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.984245 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c"] Oct 31 00:15:08.986451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:08.986435 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.024467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.024434 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd2p\" (UniqueName: \"kubernetes.io/projected/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-kube-api-access-zhd2p\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.024641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.024477 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.044132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.044105 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:15:09.044299 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.044251 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rm2xm\"" Oct 31 00:15:09.049033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.049010 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c"] Oct 31 00:15:09.053450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.053422 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:09.125085 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.125003 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd2p\" (UniqueName: \"kubernetes.io/projected/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-kube-api-access-zhd2p\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.125085 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.125039 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.127683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.127651 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.186034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.186002 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd2p\" (UniqueName: \"kubernetes.io/projected/d4b2dd2a-fd57-4667-ad81-b205b7bacbcc-kube-api-access-zhd2p\") pod \"monitoring-plugin-77f4f76594-j7f6c\" (UID: \"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.298538 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.298489 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:09.305293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.305226 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:09.687455 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:15:09.687414 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b2dd2a_fd57_4667_ad81_b205b7bacbcc.slice/crio-11da3d75b1b3b72dc67a3e4ef7b6127353df4bd2f06f0618136369d7a772f01c WatchSource:0}: Error finding container 11da3d75b1b3b72dc67a3e4ef7b6127353df4bd2f06f0618136369d7a772f01c: Status 404 returned error can't find the container with id 11da3d75b1b3b72dc67a3e4ef7b6127353df4bd2f06f0618136369d7a772f01c Oct 31 00:15:09.688207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.688182 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c"] Oct 31 00:15:09.865739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.865701 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:09.868830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.868809 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.929184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.929144 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:15:09.929376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.929247 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:15:09.929571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.929556 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:15:09.929911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.929890 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:15:09.930983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.930964 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:15:09.932056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932038 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932067 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932090 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932152 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932288 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932335 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932314 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv47\" (UniqueName: \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.932374 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.932359 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:09.978934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.978846 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:15:09.979093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.979005 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:15:09.979992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:09.979966 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:10.031084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.031057 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpkbr\"" Oct 31 00:15:10.033365 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033337 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033378 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033435 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033484 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033507 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv47\" (UniqueName: \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033550 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.033956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.033935 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.034268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.034235 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.034645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.034620 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.034732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.034662 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.036345 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.036324 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.036447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.036425 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.095956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.095924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv47\" (UniqueName: \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\") pod \"console-5fdc5bf444-tp4jb\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.170579 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.170544 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:10.179513 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.179491 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:10.319975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.319934 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:10.390998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.390963 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" event={"ID":"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc","Type":"ContainerStarted","Data":"11da3d75b1b3b72dc67a3e4ef7b6127353df4bd2f06f0618136369d7a772f01c"} Oct 31 00:15:10.569706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:10.569673 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:10.572581 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:15:10.572548 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd427f550_13f3_4fe4_af32_445e3858783d.slice/crio-3cfebe3e772d6e058475ce58a218b217d6d67aab3d87c9cc7cfa87c063281a83 WatchSource:0}: Error finding container 3cfebe3e772d6e058475ce58a218b217d6d67aab3d87c9cc7cfa87c063281a83: Status 404 returned error can't find the container with id 3cfebe3e772d6e058475ce58a218b217d6d67aab3d87c9cc7cfa87c063281a83 Oct 31 00:15:11.395285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:11.395148 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5bf444-tp4jb" event={"ID":"d427f550-13f3-4fe4-af32-445e3858783d","Type":"ContainerStarted","Data":"3cfebe3e772d6e058475ce58a218b217d6d67aab3d87c9cc7cfa87c063281a83"} Oct 31 00:15:11.397479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:11.397442 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"9cc08d0610aa231fe702e3f956c77c3e20a31634ce89eb38f269b323db62a228"} Oct 31 00:15:11.397596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:11.397504 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"259624bcedd8eba3feef7ff691bdd315ad23fcca7ea587d625ff21827259f4ed"} Oct 31 00:15:11.397596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:11.397548 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"70f42784731866bf37cb74954fd73c709a7203d5f98095f614f8876aa779305d"} Oct 31 00:15:12.833750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:12.833712 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:12.978090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:12.978059 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:13.289056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.288906 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:13.372028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.371999 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:15:13.375874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.375851 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.407825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.407792 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" event={"ID":"d4b2dd2a-fd57-4667-ad81-b205b7bacbcc","Type":"ContainerStarted","Data":"18fa1519449662ba021f85a20bfd225c11b413c2acc2a0814ae08841211715e3"} Oct 31 00:15:13.407992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.407853 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:13.411394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.411254 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"9338b861310cf371dee2869c3bb283fef647fa4fac80d2f446e05fe62beaee17"} Oct 31 00:15:13.411394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.411315 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"3ce914490f84a24aee88f13948f2a99a21c56987f09d24f34c17bdc02672705c"} Oct 31 00:15:13.411394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.411354 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" event={"ID":"7afe3f17-55c3-4edd-bfce-ddf42f5230f5","Type":"ContainerStarted","Data":"933b64f3430213329aa8e9091440c52fd9916e7a1d36f5ce96ab0f01bb0c3273"} Oct 31 00:15:13.411647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.411417 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:13.414367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.414341 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" Oct 31 00:15:13.428780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.428757 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:15:13.428893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.428789 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:15:13.428943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.428918 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:15:13.429026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.429005 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:15:13.429026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.429019 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:15:13.429139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.429062 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lg4zr\"" Oct 31 00:15:13.477178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477139 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477184 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477223 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477261 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477284 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477310 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477335 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477359 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477432 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477460 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477489 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477506 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477524 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477561 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477616 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.477642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477639 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.478050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477660 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.478050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477682 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.478050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477714 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wf59\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.478050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.477741 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.479292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.478796 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:15:13.479292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.478822 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:15:13.479292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.478833 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:15:13.479292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.478890 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:15:13.479292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.478958 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:15:13.529767 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.529736 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:15:13.578978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.578893 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.578978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.578955 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.578988 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579010 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579045 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579070 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579096 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579141 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579172 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579195 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579219 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579243 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579265 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579295 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579322 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579345 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579370 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579400 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wf59\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579456 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.579810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.579493 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.580219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.580043 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.580580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.580522 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:15:13.580673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.580641 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.581040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.581021 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-etn599rgukkmv\"" Oct 31 00:15:13.581255 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.581172 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.581331 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.581298 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:15:13.582116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.582092 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.582213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.582089 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.582510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.582487 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.582697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.582674 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.582911 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.583025 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.583040 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.583085 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.583297 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.583926 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.584315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.584127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.586855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.586315 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.586855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.586807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.587017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.586951 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.587693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.587672 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.588638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.588620 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.591541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.591504 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.592712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.592692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.592896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.592878 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.647096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.647062 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wf59\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\") pod \"prometheus-k8s-0\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.698657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.698580 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j7f6c" podStartSLOduration=2.5349622800000002 podStartE2EDuration="5.698560211s" totalImagesPullingTime="3.163597931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:13.697724109 +0000 UTC m=+40.622993062" watchObservedRunningTime="2026-10-31 00:15:13.698560211 +0000 UTC m=+40.623829163" Oct 31 00:15:13.701748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.701719 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:13.828376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:13.828303 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" podStartSLOduration=3.090420891 podStartE2EDuration="7.828283963s" totalImagesPullingTime="4.737863072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:13.820829385 +0000 UTC m=+40.746098340" watchObservedRunningTime="2026-10-31 00:15:13.828283963 +0000 UTC m=+40.753552922" Oct 31 00:15:14.139207 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:15:14.139178 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb917ad2_4a61_466e_bef5_bff5cc5028b8.slice/crio-12050a20f1e9edb1cc1cb1ba45e0a042b670ba249dad26550bc0a298c52e403f WatchSource:0}: Error finding container 12050a20f1e9edb1cc1cb1ba45e0a042b670ba249dad26550bc0a298c52e403f: Status 404 returned error can't find the container with id 12050a20f1e9edb1cc1cb1ba45e0a042b670ba249dad26550bc0a298c52e403f Oct 31 00:15:14.139660 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:14.139627 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:15:14.417456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:14.417418 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"12050a20f1e9edb1cc1cb1ba45e0a042b670ba249dad26550bc0a298c52e403f"} Oct 31 00:15:15.422599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:15.422561 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5bf444-tp4jb" event={"ID":"d427f550-13f3-4fe4-af32-445e3858783d","Type":"ContainerStarted","Data":"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744"} Oct 31 00:15:15.442654 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:15.442621 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:15.540439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:15.540392 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fdc5bf444-tp4jb" podStartSLOduration=2.696793488 podStartE2EDuration="6.540377776s" totalImagesPullingTime="3.843584288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:15.539628549 +0000 UTC m=+42.464897514" watchObservedRunningTime="2026-10-31 00:15:15.540377776 +0000 UTC m=+42.465646727" Oct 31 00:15:15.589837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:15.589794 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:15.895840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:15.895811 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.126799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:16.126761 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:16.427419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:16.427382 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" exitCode=0 Oct 31 00:15:16.427931 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:16.427471 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} Oct 31 00:15:18.194189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:18.194154 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:18.364639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:18.364600 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:19.425020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:19.424993 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fb76784f8-nc45q" Oct 31 00:15:20.180332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.180290 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:20.180565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.180361 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:20.185499 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.185473 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:20.446256 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.446176 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:20.639036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.639007 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:20.781837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:20.781752 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:21.448159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448123 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} Oct 31 00:15:21.448615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448200 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} Oct 31 00:15:21.448615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448229 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} Oct 31 00:15:21.448615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448254 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} Oct 31 00:15:21.448615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448282 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} Oct 31 00:15:21.448615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.448306 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerStarted","Data":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} Oct 31 00:15:21.657189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:21.657133 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.352365441 podStartE2EDuration="8.657119633s" totalImagesPullingTime="6.284927231s" totalInitContainerRuntime="19.826961ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:21.654411654 +0000 UTC m=+48.579680607" watchObservedRunningTime="2026-10-31 00:15:21.657119633 +0000 UTC m=+48.582388587" Oct 31 00:15:23.038072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:23.038035 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.184699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:23.184642 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.702325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:23.702288 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:15:27.633589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:27.633545 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:27.777492 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:27.777459 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:31.983854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:31.983815 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:34.281285 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:15:34.281244 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:15:38.095859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:38.095820 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:38.265571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:38.265515 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:57.003967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.003905 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fdc5bf444-tp4jb" podUID="d427f550-13f3-4fe4-af32-445e3858783d" containerName="console" containerID="cri-o://afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744" gracePeriod=15 Oct 31 00:15:57.270393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.270369 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:57.317189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317156 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317236 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317278 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\" (UniqueName: \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317311 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317354 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317381 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317408 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\") pod \"d427f550-13f3-4fe4-af32-445e3858783d\" (UID: \"d427f550-13f3-4fe4-af32-445e3858783d\") " Oct 31 00:15:57.317675 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317587 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:15:57.317675 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317665 4471 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d427f550-13f3-4fe4-af32-445e3858783d-tmp\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.317859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317806 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:57.317859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317823 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:57.317859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.317834 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:57.319476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.319456 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:57.319560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.319475 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:15:57.319608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.319568 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47" pod "d427f550-13f3-4fe4-af32-445e3858783d" (UID: "d427f550-13f3-4fe4-af32-445e3858783d"). InnerVolumeSpecName "kube-api-access-6nv47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:57.418263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418233 4471 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-console-config\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.418263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418259 4471 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.418263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418268 4471 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-serving-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.418482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418277 4471 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427f550-13f3-4fe4-af32-445e3858783d-console-oauth-config\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.418482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418286 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nv47\" (UniqueName: \"kubernetes.io/projected/d427f550-13f3-4fe4-af32-445e3858783d-kube-api-access-6nv47\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.418482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.418296 4471 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427f550-13f3-4fe4-af32-445e3858783d-oauth-serving-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:57.576056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.575961 4471 generic.go:356] "Generic (PLEG): container finished" podID="d427f550-13f3-4fe4-af32-445e3858783d" containerID="afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744" exitCode=0 Oct 31 00:15:57.576056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.576005 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdc5bf444-tp4jb" Oct 31 00:15:57.576225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.576051 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5bf444-tp4jb" event={"ID":"d427f550-13f3-4fe4-af32-445e3858783d","Type":"ContainerDied","Data":"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744"} Oct 31 00:15:57.576225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.576115 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdc5bf444-tp4jb" event={"ID":"d427f550-13f3-4fe4-af32-445e3858783d","Type":"ContainerDied","Data":"3cfebe3e772d6e058475ce58a218b217d6d67aab3d87c9cc7cfa87c063281a83"} Oct 31 00:15:57.581318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.576396 4471 scope.go:122] "RemoveContainer" containerID="afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744" Oct 31 00:15:57.590623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.590601 4471 scope.go:122] "RemoveContainer" containerID="afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744" Oct 31 00:15:57.590928 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:15:57.590898 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744\": container with ID starting with afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744 not found: ID does not exist" containerID="afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744" Oct 31 00:15:57.590977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.590941 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744"} err="failed to get container status \"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744\": rpc error: code = NotFound desc = could not find container \"afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744\": container with ID starting with afd41b5512d67a05a81004d88f0f1a1cebd8b937e63e0d73e84616b6b93a0744 not found: ID does not exist" Oct 31 00:15:57.739848 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.739810 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:57.743053 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:57.743033 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdc5bf444-tp4jb"] Oct 31 00:15:59.206194 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:15:59.206150 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d427f550-13f3-4fe4-af32-445e3858783d" path="/var/lib/kubelet/pods/d427f550-13f3-4fe4-af32-445e3858783d/volumes" Oct 31 00:16:04.279040 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:04.279002 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:16:16.146580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:16.146543 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:16.434467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:16.434392 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:19.598617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:19.598574 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:19.617263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:19.617240 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:19.666868 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:19.666843 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:27.913458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:27.913425 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.119684 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:28.119602 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.279519 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:34.279483 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:16:35.850009 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.849967 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850375 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="prometheus" containerID="cri-o://945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" gracePeriod=600 Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850380 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="thanos-sidecar" containerID="cri-o://723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" gracePeriod=600 Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850417 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" gracePeriod=600 Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850508 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy" containerID="cri-o://38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" gracePeriod=600 Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850505 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-web" containerID="cri-o://6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" gracePeriod=600 Oct 31 00:16:35.850616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:35.850551 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="config-reloader" containerID="cri-o://6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" gracePeriod=600 Oct 31 00:16:36.106213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.106156 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:36.194423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194396 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194431 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194450 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194469 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194485 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194505 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194695 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194746 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194788 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.194841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194825 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194859 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194902 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194868 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194961 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194990 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.195037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.194995 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195049 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195271 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.195333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195279 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195083 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195604 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195638 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195669 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195675 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.195912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195763 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\" (UID: \"fb917ad2-4a61-466e-bef5-bff5cc5028b8\") " Oct 31 00:16:36.196293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195967 4471 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.196293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.195986 4471 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.196293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196000 4471 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-metrics-client-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.196293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196015 4471 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.196293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196031 4471 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.196555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196440 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.196888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196862 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:36.196976 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.196909 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:36.197335 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.197308 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.198005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.197979 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:36.198332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.198307 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "kube-api-access-4wf59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:36.198579 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.198550 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.199056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199032 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.199056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199039 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:16:36.199220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199196 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.199343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199273 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.199541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199505 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.199812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.199792 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.200435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.200418 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.208210 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.208192 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config" pod "fb917ad2-4a61-466e-bef5-bff5cc5028b8" (UID: "fb917ad2-4a61-466e-bef5-bff5cc5028b8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296844 4471 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-db\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296878 4471 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config-out\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296894 4471 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296911 4471 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296928 4471 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-config\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296942 4471 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296958 4471 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-kube-rbac-proxy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.296978 4471 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-grpc-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297010 4471 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-tls-assets\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297035 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wf59\" (UniqueName: \"kubernetes.io/projected/fb917ad2-4a61-466e-bef5-bff5cc5028b8-kube-api-access-4wf59\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297052 4471 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297061 4471 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297070 4471 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb917ad2-4a61-466e-bef5-bff5cc5028b8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297078 4471 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-secret-metrics-client-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.297409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.297087 4471 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb917ad2-4a61-466e-bef5-bff5cc5028b8-web-config\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:36.710046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710010 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" exitCode=0 Oct 31 00:16:36.710046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710039 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" exitCode=0 Oct 31 00:16:36.710046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710045 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" exitCode=0 Oct 31 00:16:36.710046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710051 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" exitCode=0 Oct 31 00:16:36.710046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710057 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" exitCode=0 Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710063 4471 generic.go:356] "Generic (PLEG): container finished" podID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" exitCode=0 Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710084 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710089 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710143 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710175 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710207 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710221 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710245 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710283 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} Oct 31 00:16:36.710443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.710322 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb917ad2-4a61-466e-bef5-bff5cc5028b8","Type":"ContainerDied","Data":"12050a20f1e9edb1cc1cb1ba45e0a042b670ba249dad26550bc0a298c52e403f"} Oct 31 00:16:36.720092 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.720075 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.727878 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.727853 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.735043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.735026 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.742072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.742051 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.751282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.750182 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.758920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.758899 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.765759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.765738 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.766020 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.766002 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.766094 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766035 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.766094 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766061 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.766308 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.766290 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.766350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766315 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.766350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766331 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.766610 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.766594 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.766667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766616 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.766667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766636 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.766874 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.766857 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.766910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766879 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.766910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.766892 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.767111 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.767094 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.767151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767115 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.767151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767130 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.767329 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.767314 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.767384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767337 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.767384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767358 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.767605 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:16:36.767589 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.767647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767609 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.767647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767622 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.767850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767832 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.767918 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.767851 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.768060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768044 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.768104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768060 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.768278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768260 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.768343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768279 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.768493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768475 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.768561 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768494 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.768734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768718 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.768783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768735 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.768932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768914 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.769000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.768933 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.769175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769159 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.769219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769175 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.769368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769351 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.769408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769368 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.769596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769579 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.769646 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769596 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.769783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769766 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.769849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769784 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.769974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769954 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.770010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.769974 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.770157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770142 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.770222 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770158 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.770387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770369 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.770431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770388 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.770596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770580 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.770657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770596 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.770809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770793 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.770856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770810 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.771001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.770986 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.771001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771000 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.771174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771159 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.771214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771173 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.771383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771367 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.771456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771384 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.771616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771601 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.771663 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771616 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.771804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771782 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.771840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771805 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.771992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771974 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.772052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.771993 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.772196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772180 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.772240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772197 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.772376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772360 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.772442 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772378 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.772666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772651 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.772715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772667 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.772866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772851 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.772929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.772867 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.773072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773052 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.773151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773073 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.773261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773244 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.773325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773262 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.773461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773446 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.773506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773460 4471 scope.go:122] "RemoveContainer" containerID="fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d" Oct 31 00:16:36.773686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773669 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d"} err="failed to get container status \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": rpc error: code = NotFound desc = could not find container \"fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d\": container with ID starting with fd6bf0005047121fac219ab5f16531641573746b8fa3422e63dcfc1aba853a1d not found: ID does not exist" Oct 31 00:16:36.773760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773687 4471 scope.go:122] "RemoveContainer" containerID="38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f" Oct 31 00:16:36.773885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773866 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f"} err="failed to get container status \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": rpc error: code = NotFound desc = could not find container \"38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f\": container with ID starting with 38bbf1b30cad90cdd974cd5a3acf8d88fc7f104f0ca97118508af03c0e84d30f not found: ID does not exist" Oct 31 00:16:36.773928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.773886 4471 scope.go:122] "RemoveContainer" containerID="6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4" Oct 31 00:16:36.774071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774055 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4"} err="failed to get container status \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": rpc error: code = NotFound desc = could not find container \"6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4\": container with ID starting with 6dddcbd4e8bac988d0084f4ad56496b9ce60aed9f084c31e36bc2f3d291266f4 not found: ID does not exist" Oct 31 00:16:36.774113 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774071 4471 scope.go:122] "RemoveContainer" containerID="723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5" Oct 31 00:16:36.774245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774227 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5"} err="failed to get container status \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": rpc error: code = NotFound desc = could not find container \"723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5\": container with ID starting with 723b2d2f8d6c9a83e1f7db29165047b7f9ea6eb42107217449443f13ed5edaa5 not found: ID does not exist" Oct 31 00:16:36.774312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774246 4471 scope.go:122] "RemoveContainer" containerID="6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135" Oct 31 00:16:36.774443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774428 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135"} err="failed to get container status \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": rpc error: code = NotFound desc = could not find container \"6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135\": container with ID starting with 6a241a7a0ba72d81a9b3fa6ee7d2731afea8677b1fcc5af1a23c5eac00585135 not found: ID does not exist" Oct 31 00:16:36.774483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774442 4471 scope.go:122] "RemoveContainer" containerID="945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e" Oct 31 00:16:36.774659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774642 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e"} err="failed to get container status \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": rpc error: code = NotFound desc = could not find container \"945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e\": container with ID starting with 945c71964b069afba31930745972e08b97cf9dd1978e6b15e8675080a6ebbd4e not found: ID does not exist" Oct 31 00:16:36.774727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774660 4471 scope.go:122] "RemoveContainer" containerID="02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6" Oct 31 00:16:36.774860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.774844 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6"} err="failed to get container status \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": rpc error: code = NotFound desc = could not find container \"02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6\": container with ID starting with 02de00baeb4289daf5d99bc3f3857db05e4de9034176bcf35bb185832eebbde6 not found: ID does not exist" Oct 31 00:16:36.878365 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.878343 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:36.928326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.928301 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:36.928956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.928937 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:36.929260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929244 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="thanos-sidecar" Oct 31 00:16:36.929318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929261 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="thanos-sidecar" Oct 31 00:16:36.929318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929275 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="config-reloader" Oct 31 00:16:36.929318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929282 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="config-reloader" Oct 31 00:16:36.929318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929298 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-web" Oct 31 00:16:36.929318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929303 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-web" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929341 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="init-config-reloader" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929347 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="init-config-reloader" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929354 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d427f550-13f3-4fe4-af32-445e3858783d" containerName="console" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929358 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d427f550-13f3-4fe4-af32-445e3858783d" containerName="console" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929365 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929369 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929379 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929384 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929395 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="prometheus" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929402 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="prometheus" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929461 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-web" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929468 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929476 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d427f550-13f3-4fe4-af32-445e3858783d" containerName="console" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929485 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="kube-rbac-proxy-thanos" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929493 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="prometheus" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929499 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="thanos-sidecar" Oct 31 00:16:36.929502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.929504 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" containerName="config-reloader" Oct 31 00:16:36.934673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.934657 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:36.997850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.997781 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:16:36.997850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.997821 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:16:36.998041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.997979 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:16:36.998041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998026 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:16:36.998144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998037 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lg4zr\"" Oct 31 00:16:36.998235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998221 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:16:36.998306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998222 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-etn599rgukkmv\"" Oct 31 00:16:36.998306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998226 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:16:36.998306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998260 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:16:36.998306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998260 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:16:36.998490 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998428 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:16:36.998958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998938 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:16:36.998958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998939 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:16:36.999106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998957 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:16:36.999106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:36.998966 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:16:37.049536 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.049506 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:16:37.049663 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.049648 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:16:37.095324 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.095300 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:37.102288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102268 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102295 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102313 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102333 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102352 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102459 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102493 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102596 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102622 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102638 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102664 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102732 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102755 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102780 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnth\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-kube-api-access-6fnth\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102799 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102837 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102884 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.102919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102902 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.103002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102917 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.103002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.102933 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203649 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203690 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203707 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203722 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnth\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-kube-api-access-6fnth\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203739 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203763 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.203802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203799 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203822 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203849 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203880 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203913 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203935 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203951 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203974 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.203996 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204020 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204041 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204068 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204097 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.204124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204123 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.205088 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.204552 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.206026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.205462 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.206026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.205731 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.206929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.206908 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.207402 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb917ad2-4a61-466e-bef5-bff5cc5028b8" path="/var/lib/kubelet/pods/fb917ad2-4a61-466e-bef5-bff5cc5028b8/volumes" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.207442 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.207920 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.208443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.208798 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.208880 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.209014 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.209497 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.209514 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.209642 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.209992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.209917 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.210240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.210018 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.210298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.210251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.210373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.210349 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.210428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.210410 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.210780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.210766 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.264825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.264775 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnth\" (UniqueName: \"kubernetes.io/projected/b8fa41aa-7f57-41b1-b056-4bcfd23356c4-kube-api-access-6fnth\") pod \"prometheus-k8s-0\" (UID: \"b8fa41aa-7f57-41b1-b056-4bcfd23356c4\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:37.558237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:37.558155 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:38.029352 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:16:38.029311 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fa41aa_7f57_41b1_b056_4bcfd23356c4.slice/crio-1c3d3fe39ad5368469ce3eba0149c334fef0621af59e93fc7da0ce6329c7e09b WatchSource:0}: Error finding container 1c3d3fe39ad5368469ce3eba0149c334fef0621af59e93fc7da0ce6329c7e09b: Status 404 returned error can't find the container with id 1c3d3fe39ad5368469ce3eba0149c334fef0621af59e93fc7da0ce6329c7e09b Oct 31 00:16:38.029992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:38.029969 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:16:38.718580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:38.718546 4471 generic.go:356] "Generic (PLEG): container finished" podID="b8fa41aa-7f57-41b1-b056-4bcfd23356c4" containerID="7eb131c5542ab85ab11fdcf49daeada36ebd415105577a84adf5cf6041daeeca" exitCode=0 Oct 31 00:16:38.718737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:38.718584 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerDied","Data":"7eb131c5542ab85ab11fdcf49daeada36ebd415105577a84adf5cf6041daeeca"} Oct 31 00:16:38.718737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:38.718634 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"1c3d3fe39ad5368469ce3eba0149c334fef0621af59e93fc7da0ce6329c7e09b"} Oct 31 00:16:39.057678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.057640 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.278159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.278084 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.430747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.430716 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.578682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.578605 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:39.730408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730374 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"789cd2b7c0217b42163e091831eb30d2bdd2d51488e0e81b5808f6ad9d5bdc47"} Oct 31 00:16:39.730597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730457 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"9c4996ebf501777ef5b4b9e16b2d64c4b19b77f407ad7d0f7e684fe12cad17b7"} Oct 31 00:16:39.730597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730483 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"8734814f37b2edcc99761d8b4f0352a4dc3090d34c25a05022091b888e72222e"} Oct 31 00:16:39.730597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730506 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"298a70a3ce2b6d2e4fee1412f72117193a464e0868fc9b91511679eca2fa17fc"} Oct 31 00:16:39.730597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730549 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"ffb97ffec5c776a19a7d57ba379d4bc856b3423335d519f0d451f715efc51d7f"} Oct 31 00:16:39.730597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.730578 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b8fa41aa-7f57-41b1-b056-4bcfd23356c4","Type":"ContainerStarted","Data":"471064c4463a08394b95f3d196195b56a6d96edd4f33805a03c97d3e9a01197e"} Oct 31 00:16:39.863238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:39.863154 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.847213814 podStartE2EDuration="3.863143067s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.929253ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:39.862513118 +0000 UTC m=+126.787782072" watchObservedRunningTime="2026-10-31 00:16:39.863143067 +0000 UTC m=+126.788412020" Oct 31 00:16:42.559218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:42.559170 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:16:45.977678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:16:45.977594 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:04.278635 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:17:04.278597 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:17:16.482922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:16.482885 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:34.278515 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:17:34.278477 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:17:37.559353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:37.559321 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:17:37.574472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:37.574447 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:17:37.938466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:37.938443 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:17:45.790489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:45.790447 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:45.950026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:45.949989 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.264193 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:48.264166 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.415008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:48.414979 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.669085 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:48.669053 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:48.813273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:48.813240 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:58.395109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:58.395077 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:58.542914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:17:58.542873 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:04.279046 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:18:04.279010 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:18:08.064012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:18:08.063980 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:08.225743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:18:08.225713 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:16.819735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:18:16.819644 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:34.279967 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:18:34.279924 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:19:01.542543 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:01.542498 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:04.277911 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:19:04.277875 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:19:17.150893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:17.150860 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:18.388999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:18.388965 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:34.278015 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:19:34.277977 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:19:49.975268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:49.975193 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:50.215125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:50.215090 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.673280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:51.673244 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:51.826740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:51.826704 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:53.010830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:53.010796 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:58.628436 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:58.628404 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:58.772664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:58.772634 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.151133 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.151105 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.385856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.385818 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj"] Oct 31 00:19:59.389036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.389015 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.396869 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.396844 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.471845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.471808 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.471998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.471852 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.471998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.471872 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.471998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.471927 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472036 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472058 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472092 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qnl\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-kube-api-access-z5qnl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472190 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472131 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472224 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.472914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472898 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:19:59.472973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472914 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:19:59.472973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.472904 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:19:59.488164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.488145 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6pnc8\"" Oct 31 00:19:59.489293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.489274 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj"] Oct 31 00:19:59.550320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.550262 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:59.572541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572500 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572563 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572587 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572602 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572625 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572797 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572645 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572811 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qnl\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-kube-api-access-z5qnl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572865 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.572927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.572909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.573092 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.573041 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.573199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.573113 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.573199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.573155 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.573325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.573280 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.573549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.573514 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.574982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.574960 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.575250 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.575234 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.630785 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.630749 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.630785 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.630779 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qnl\" (UniqueName: \"kubernetes.io/projected/6c2bbcde-ca2d-493b-8c69-8a81711e5faf-kube-api-access-z5qnl\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj\" (UID: \"6c2bbcde-ca2d-493b-8c69-8a81711e5faf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:19:59.702041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:19:59.701952 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:20:00.120454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:00.120366 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj"] Oct 31 00:20:00.147255 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:20:00.147227 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2bbcde_ca2d_493b_8c69_8a81711e5faf.slice/crio-73bf4cb3714016045e7b9c93757f851c5ee3e1579f31b7f172afff9f994ffbd4 WatchSource:0}: Error finding container 73bf4cb3714016045e7b9c93757f851c5ee3e1579f31b7f172afff9f994ffbd4: Status 404 returned error can't find the container with id 73bf4cb3714016045e7b9c93757f851c5ee3e1579f31b7f172afff9f994ffbd4 Oct 31 00:20:00.149302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:00.149288 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:20:00.397721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:00.397693 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" event={"ID":"6c2bbcde-ca2d-493b-8c69-8a81711e5faf","Type":"ContainerStarted","Data":"73bf4cb3714016045e7b9c93757f851c5ee3e1579f31b7f172afff9f994ffbd4"} Oct 31 00:20:02.209136 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:02.209082 4471 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/d4/d401fd69bdddf38670ec84be70c2da90a65270bc7595a1460d76ff52b1a25bea?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T002000Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=49c7087d5f335e5f2eec4084ea08eeb6263e9ae139274f7ef76fe74d722c44c2®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793406900~hmac=c9f80f77cda8e66576ab022a8261764a6cdd544d6d47435885ea581de7e8e510\": EOF" image="registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55" Oct 31 00:20:02.209623 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:02.209315 4471 kuberuntime_manager.go:1707] "Unhandled Error" err="container istio-proxy start failed in pod openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj_openshift-ingress(6c2bbcde-ca2d-493b-8c69-8a81711e5faf): ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/d4/d401fd69bdddf38670ec84be70c2da90a65270bc7595a1460d76ff52b1a25bea?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T002000Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=49c7087d5f335e5f2eec4084ea08eeb6263e9ae139274f7ef76fe74d722c44c2®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793406900~hmac=c9f80f77cda8e66576ab022a8261764a6cdd544d6d47435885ea581de7e8e510\": EOF" logger="UnhandledError" Oct 31 00:20:02.209623 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:02.209354 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ErrImagePull: \"unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/d4/d401fd69bdddf38670ec84be70c2da90a65270bc7595a1460d76ff52b1a25bea?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T002000Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=49c7087d5f335e5f2eec4084ea08eeb6263e9ae139274f7ef76fe74d722c44c2®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793406900~hmac=c9f80f77cda8e66576ab022a8261764a6cdd544d6d47435885ea581de7e8e510\\\": EOF\"" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" podUID="6c2bbcde-ca2d-493b-8c69-8a81711e5faf" Oct 31 00:20:02.407695 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:02.407648 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55\\\": ErrImagePull: unable to pull image: copying system image from manifest list: parsing image configuration: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/d4/d401fd69bdddf38670ec84be70c2da90a65270bc7595a1460d76ff52b1a25bea?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261031%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261031T002000Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=49c7087d5f335e5f2eec4084ea08eeb6263e9ae139274f7ef76fe74d722c44c2®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=openshift-service-mesh----istio-proxyv2-rhel9&akamai_signature=exp=1793406900~hmac=c9f80f77cda8e66576ab022a8261764a6cdd544d6d47435885ea581de7e8e510\\\": EOF\"" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" podUID="6c2bbcde-ca2d-493b-8c69-8a81711e5faf" Oct 31 00:20:03.153053 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:03.153010 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:04.279749 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:04.279714 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:20:17.518920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:17.518853 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:17.583158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:17.583129 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:20:17.583239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:17.583202 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:20:17.583239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:17.583234 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:20:18.461818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:18.461781 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" event={"ID":"6c2bbcde-ca2d-493b-8c69-8a81711e5faf","Type":"ContainerStarted","Data":"0fe9d391f15cc2f1f741c3a7facfa6cf3515acf7f011f5e59a5c37305b56dee7"} Oct 31 00:20:18.622383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:18.622288 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" podStartSLOduration=2.189475114 podStartE2EDuration="19.622272242s" totalImagesPullingTime="17.432797128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-31 00:20:18.621444275 +0000 UTC m=+345.546713226" watchObservedRunningTime="2026-10-31 00:20:18.622272242 +0000 UTC m=+345.547541196" Oct 31 00:20:18.703154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:18.703117 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:20:18.707401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:18.707374 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:20:19.465793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:19.465759 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:20:19.466735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:19.466713 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj" Oct 31 00:20:22.936196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:22.936164 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:23.083095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:23.083062 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:26.490716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:26.490687 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:26.669047 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:26.668992 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.174906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.174871 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.417583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.417551 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49"] Oct 31 00:20:27.420862 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.420845 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.478360 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478325 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:20:27.478678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478407 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:20:27.478678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478486 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:20:27.478678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478554 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:20:27.478884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478723 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:20:27.478884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.478781 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vk4k4\"" Oct 31 00:20:27.508690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508662 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.509045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508713 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k556s\" (UniqueName: \"kubernetes.io/projected/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-kube-api-access-k556s\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.509045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508788 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.509045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508832 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.509045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508864 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.509045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.508887 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.526592 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.526563 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49"] Oct 31 00:20:27.540354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.540326 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.609501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609467 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.609691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609563 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k556s\" (UniqueName: \"kubernetes.io/projected/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-kube-api-access-k556s\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.609691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609591 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.609691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.609691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609633 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.609691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.609669 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.610224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.610191 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-manifests\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.611980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.611958 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-tmp\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.612241 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.612219 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.612443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.612427 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.612604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.612584 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.667071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.667034 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k556s\" (UniqueName: \"kubernetes.io/projected/57c79a12-7f1e-4008-aae0-a9fbd35e0a39-kube-api-access-k556s\") pod \"opendatahub-operator-controller-manager-7fbb8789b-9cf49\" (UID: \"57c79a12-7f1e-4008-aae0-a9fbd35e0a39\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.699273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.699241 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:27.735412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.735380 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:27.857887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:27.857851 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:28.234842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:28.234816 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49"] Oct 31 00:20:28.236001 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:20:28.235962 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c79a12_7f1e_4008_aae0_a9fbd35e0a39.slice/crio-262a0e28fb49bc854b7acdbed44c7b97f0b9275625432ef3fa8cee37b9a77024 WatchSource:0}: Error finding container 262a0e28fb49bc854b7acdbed44c7b97f0b9275625432ef3fa8cee37b9a77024: Status 404 returned error can't find the container with id 262a0e28fb49bc854b7acdbed44c7b97f0b9275625432ef3fa8cee37b9a77024 Oct 31 00:20:28.452387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:28.452355 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:28.497172 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:28.497092 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" event={"ID":"57c79a12-7f1e-4008-aae0-a9fbd35e0a39","Type":"ContainerStarted","Data":"262a0e28fb49bc854b7acdbed44c7b97f0b9275625432ef3fa8cee37b9a77024"} Oct 31 00:20:30.317066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:30.317026 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:30.476174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:30.476138 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.459885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:31.459841 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:31.616324 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:31.616281 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:32.512739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:32.512654 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" event={"ID":"57c79a12-7f1e-4008-aae0-a9fbd35e0a39","Type":"ContainerStarted","Data":"f86486b4464e292003ad7459fd07914cb99f48db7475841221361a382322baa5"} Oct 31 00:20:33.518204 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:33.518166 4471 generic.go:356] "Generic (PLEG): container finished" podID="57c79a12-7f1e-4008-aae0-a9fbd35e0a39" containerID="f86486b4464e292003ad7459fd07914cb99f48db7475841221361a382322baa5" exitCode=0 Oct 31 00:20:33.518710 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:33.518226 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" event={"ID":"57c79a12-7f1e-4008-aae0-a9fbd35e0a39","Type":"ContainerDied","Data":"f86486b4464e292003ad7459fd07914cb99f48db7475841221361a382322baa5"} Oct 31 00:20:34.278498 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:20:34.278461 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:20:34.785127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:34.785046 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:34.959319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:34.959286 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.526423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:35.526383 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" event={"ID":"57c79a12-7f1e-4008-aae0-a9fbd35e0a39","Type":"ContainerStarted","Data":"c3ab6fad111a4a52ce00410824127168e6d44a51527119074e400f3d43049ba5"} Oct 31 00:20:35.526650 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:35.526516 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:35.695607 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:35.695572 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:35.760779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:35.760712 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" podStartSLOduration=3.337437744 podStartE2EDuration="8.760697795s" totalImagesPullingTime="4.927624002s" totalInitContainerRuntime="495.636049ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:35.714636604 +0000 UTC m=+362.639905575" watchObservedRunningTime="2026-10-31 00:20:35.760697795 +0000 UTC m=+362.685966748" Oct 31 00:20:36.878028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:36.877968 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:38.115678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:38.115642 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:40.556341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:40.556305 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:40.722029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:40.721990 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:43.089014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.088977 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v"] Oct 31 00:20:43.094788 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.094763 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.189425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.189386 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:20:43.189425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.189426 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:20:43.189650 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.189553 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dnrk6\"" Oct 31 00:20:43.190613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.190579 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v"] Oct 31 00:20:43.237750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.237722 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgks\" (UniqueName: \"kubernetes.io/projected/30bc8157-8802-494c-a5ad-389430e88917-kube-api-access-zkgks\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.237750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.237753 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30bc8157-8802-494c-a5ad-389430e88917-tmp\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.338405 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.338368 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgks\" (UniqueName: \"kubernetes.io/projected/30bc8157-8802-494c-a5ad-389430e88917-kube-api-access-zkgks\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.338600 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.338412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30bc8157-8802-494c-a5ad-389430e88917-tmp\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.338826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.338807 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30bc8157-8802-494c-a5ad-389430e88917-tmp\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.396028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.395996 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgks\" (UniqueName: \"kubernetes.io/projected/30bc8157-8802-494c-a5ad-389430e88917-kube-api-access-zkgks\") pod \"jobset-operator-674cd59dc5-rp79v\" (UID: \"30bc8157-8802-494c-a5ad-389430e88917\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.420397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.420369 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" Oct 31 00:20:43.780403 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:43.780378 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v"] Oct 31 00:20:43.783027 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:20:43.782999 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30bc8157_8802_494c_a5ad_389430e88917.slice/crio-51f0bf390db7cbaef345c031abc050271b969bdbeb176c02ea1e5e223ff60e3b WatchSource:0}: Error finding container 51f0bf390db7cbaef345c031abc050271b969bdbeb176c02ea1e5e223ff60e3b: Status 404 returned error can't find the container with id 51f0bf390db7cbaef345c031abc050271b969bdbeb176c02ea1e5e223ff60e3b Oct 31 00:20:44.558184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:44.558146 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" event={"ID":"30bc8157-8802-494c-a5ad-389430e88917","Type":"ContainerStarted","Data":"51f0bf390db7cbaef345c031abc050271b969bdbeb176c02ea1e5e223ff60e3b"} Oct 31 00:20:44.954900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:44.954862 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:46.532570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:46.532520 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fbb8789b-9cf49" Oct 31 00:20:48.110195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:48.110138 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:48.391645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:48.391616 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:48.576205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:48.576165 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" event={"ID":"30bc8157-8802-494c-a5ad-389430e88917","Type":"ContainerStarted","Data":"2c16450d8215b73b79ec200f0a18997535eb68c5c6c1ac7fe3f63f347ff0f58f"} Oct 31 00:20:48.686999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:48.686947 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rp79v" podStartSLOduration=1.1513883950000001 podStartE2EDuration="5.686932378s" totalImagesPullingTime="4.535543983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:48.686059204 +0000 UTC m=+375.611328156" watchObservedRunningTime="2026-10-31 00:20:48.686932378 +0000 UTC m=+375.612201329" Oct 31 00:20:49.629375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:49.629339 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:50.128686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:50.128635 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:50.569461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:50.569430 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:53.734229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:53.734190 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:54.098906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:54.098834 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.352654 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:55.352623 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:55.519580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:55.519520 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.654849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.654816 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.682431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.682390 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq"] Oct 31 00:20:56.685067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.685040 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.736494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.736458 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r5wr4\"" Oct 31 00:20:56.739199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.739174 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 00:20:56.751040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.751002 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95f5fe9d-cd0b-492c-ab83-a2c90700f662-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.751040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.751045 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwdr9\" (UniqueName: \"kubernetes.io/projected/95f5fe9d-cd0b-492c-ab83-a2c90700f662-kube-api-access-gwdr9\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.751253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.751083 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95f5fe9d-cd0b-492c-ab83-a2c90700f662-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.783108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.783072 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 00:20:56.783108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.783072 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:20:56.783108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.783072 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:20:56.784356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.784333 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq"] Oct 31 00:20:56.851759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.851720 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95f5fe9d-cd0b-492c-ab83-a2c90700f662-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.851759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.851763 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwdr9\" (UniqueName: \"kubernetes.io/projected/95f5fe9d-cd0b-492c-ab83-a2c90700f662-kube-api-access-gwdr9\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.851966 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.851804 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95f5fe9d-cd0b-492c-ab83-a2c90700f662-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.852420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.852396 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95f5fe9d-cd0b-492c-ab83-a2c90700f662-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.854434 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.854409 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95f5fe9d-cd0b-492c-ab83-a2c90700f662-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.900459 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.900424 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:56.909590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.909510 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwdr9\" (UniqueName: \"kubernetes.io/projected/95f5fe9d-cd0b-492c-ab83-a2c90700f662-kube-api-access-gwdr9\") pod \"kuadrant-console-plugin-66c4fb575d-zxthq\" (UID: \"95f5fe9d-cd0b-492c-ab83-a2c90700f662\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:56.996798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:56.996602 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" Oct 31 00:20:57.360827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:57.360755 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq"] Oct 31 00:20:57.363948 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:20:57.363919 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95f5fe9d_cd0b_492c_ab83_a2c90700f662.slice/crio-767c2c6ab2aeaa247c69ecc62eda9a52591e735809f411b510bcc42ab5bd3fc0 WatchSource:0}: Error finding container 767c2c6ab2aeaa247c69ecc62eda9a52591e735809f411b510bcc42ab5bd3fc0: Status 404 returned error can't find the container with id 767c2c6ab2aeaa247c69ecc62eda9a52591e735809f411b510bcc42ab5bd3fc0 Oct 31 00:20:57.612873 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:57.612787 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" event={"ID":"95f5fe9d-cd0b-492c-ab83-a2c90700f662","Type":"ContainerStarted","Data":"767c2c6ab2aeaa247c69ecc62eda9a52591e735809f411b510bcc42ab5bd3fc0"} Oct 31 00:20:57.971312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:57.971279 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:58.139287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:20:58.139214 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:00.820046 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:00.820010 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.128357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:01.128279 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.284844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:01.284800 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.693505 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:01.693470 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:01.854687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:01.854653 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:02.085246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.085167 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v"] Oct 31 00:21:02.101912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.101878 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.158223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.156626 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v"] Oct 31 00:21:02.189420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.189382 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:21:02.189420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.189383 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:21:02.189681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.189451 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6px45\"" Oct 31 00:21:02.201906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.201878 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2bk\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-kube-api-access-fl2bk\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.202054 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.201914 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.291830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.291795 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:02.303051 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.303019 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2bk\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-kube-api-access-fl2bk\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.303220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.303065 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.360622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.360522 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.360759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.360733 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2bk\" (UniqueName: \"kubernetes.io/projected/ed2996a2-f3db-4ddf-bd16-0672659200fe-kube-api-access-fl2bk\") pod \"cert-manager-webhook-5dbbbf75cc-nck4v\" (UID: \"ed2996a2-f3db-4ddf-bd16-0672659200fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.417477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.417442 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:02.444820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.444790 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:02.821692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:02.821665 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v"] Oct 31 00:21:02.823395 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:02.823352 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded2996a2_f3db_4ddf_bd16_0672659200fe.slice/crio-21895f25b6eb080a945cef7fce6131fe70b2b19b64b89ffbf123c0c34284f5b8 WatchSource:0}: Error finding container 21895f25b6eb080a945cef7fce6131fe70b2b19b64b89ffbf123c0c34284f5b8: Status 404 returned error can't find the container with id 21895f25b6eb080a945cef7fce6131fe70b2b19b64b89ffbf123c0c34284f5b8 Oct 31 00:21:03.229483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.229441 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.639620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.639582 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" event={"ID":"ed2996a2-f3db-4ddf-bd16-0672659200fe","Type":"ContainerStarted","Data":"21895f25b6eb080a945cef7fce6131fe70b2b19b64b89ffbf123c0c34284f5b8"} Oct 31 00:21:03.641570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.641508 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" event={"ID":"95f5fe9d-cd0b-492c-ab83-a2c90700f662","Type":"ContainerStarted","Data":"8b27645325dcd5e8cd3763c16340154bdb9a098660c0be2a8895d6831084ebde"} Oct 31 00:21:03.642072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.641983 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6"] Oct 31 00:21:03.644711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.644692 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.714258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.714218 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cfjd9\"" Oct 31 00:21:03.715421 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.715398 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpz6\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-kube-api-access-vzpz6\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.715583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.715517 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.737061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.737027 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.746158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.746126 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6"] Oct 31 00:21:03.760996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.760949 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zxthq" podStartSLOduration=1.5761619709999999 podStartE2EDuration="7.760936192s" totalImagesPullingTime="6.184774221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:03.760484151 +0000 UTC m=+390.685753103" watchObservedRunningTime="2026-10-31 00:21:03.760936192 +0000 UTC m=+390.686205143" Oct 31 00:21:03.817042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.816946 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpz6\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-kube-api-access-vzpz6\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.817188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.817071 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.886057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.886022 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.886196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.886080 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpz6\" (UniqueName: \"kubernetes.io/projected/4da81e0b-257e-45dc-bbed-88ca3589a4b4-kube-api-access-vzpz6\") pod \"cert-manager-cainjector-5c57bbd4fc-zxhr6\" (UID: \"4da81e0b-257e-45dc-bbed-88ca3589a4b4\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:03.941874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.941838 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:03.954623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:03.954602 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" Oct 31 00:21:04.149463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:04.149424 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:04.278924 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:21:04.278889 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:21:04.327239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:04.327209 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6"] Oct 31 00:21:04.330345 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:04.330322 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4da81e0b_257e_45dc_bbed_88ca3589a4b4.slice/crio-9a1514ef3529c38727d7e0a190494b38d1d55c432e834f7dcf171a339ef80732 WatchSource:0}: Error finding container 9a1514ef3529c38727d7e0a190494b38d1d55c432e834f7dcf171a339ef80732: Status 404 returned error can't find the container with id 9a1514ef3529c38727d7e0a190494b38d1d55c432e834f7dcf171a339ef80732 Oct 31 00:21:04.590982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:04.590896 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:04.645964 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:04.645929 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" event={"ID":"4da81e0b-257e-45dc-bbed-88ca3589a4b4","Type":"ContainerStarted","Data":"9a1514ef3529c38727d7e0a190494b38d1d55c432e834f7dcf171a339ef80732"} Oct 31 00:21:05.931323 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:05.931287 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:06.039766 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.039735 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:06.043305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.043273 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-v9hv8"] Oct 31 00:21:06.043451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.043416 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.046104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.046080 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.100162 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.100120 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:21:06.107282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.107250 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:06.138218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.138183 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7731d80-c8a6-4db2-a068-d89a39303720-tmp\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.138389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.138246 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/d7731d80-c8a6-4db2-a068-d89a39303720-kube-api-access-qbhlj\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.138389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.138280 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5v26\" (UniqueName: \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.138389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.138317 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.138389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.138352 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7731d80-c8a6-4db2-a068-d89a39303720-tls-certs\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.144295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.144267 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-v9hv8"] Oct 31 00:21:06.173930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.173904 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r4rfq\"" Oct 31 00:21:06.174072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.173959 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:21:06.174072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.174019 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:21:06.180209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.180142 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:06.238816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.238740 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/d7731d80-c8a6-4db2-a068-d89a39303720-kube-api-access-qbhlj\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.238816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.238783 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5v26\" (UniqueName: \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.239027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.238832 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.239027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.238862 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7731d80-c8a6-4db2-a068-d89a39303720-tls-certs\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.239027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.238966 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7731d80-c8a6-4db2-a068-d89a39303720-tmp\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.239767 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.239737 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.242104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.242079 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7731d80-c8a6-4db2-a068-d89a39303720-tls-certs\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.244309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.244051 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7731d80-c8a6-4db2-a068-d89a39303720-tmp\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.302380 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.302348 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5v26\" (UniqueName: \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\") pod \"limitador-limitador-74dfcfb894-jkwrv\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.304301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.304267 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhlj\" (UniqueName: \"kubernetes.io/projected/d7731d80-c8a6-4db2-a068-d89a39303720-kube-api-access-qbhlj\") pod \"kube-auth-proxy-66d4777458-v9hv8\" (UID: \"d7731d80-c8a6-4db2-a068-d89a39303720\") " pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.311133 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.311107 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:06.363498 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.363463 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:06.371211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.371185 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" Oct 31 00:21:06.686845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.686756 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:06.744449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.744411 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:06.754077 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:06.754045 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bdf63e_eadc_4fc5_b007_0004edfcb1ad.slice/crio-a9fb47d8e6ba2820735bf5bd374e72778a03c90138593ab8a86bc0570b159baf WatchSource:0}: Error finding container a9fb47d8e6ba2820735bf5bd374e72778a03c90138593ab8a86bc0570b159baf: Status 404 returned error can't find the container with id a9fb47d8e6ba2820735bf5bd374e72778a03c90138593ab8a86bc0570b159baf Oct 31 00:21:06.791989 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.791963 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66d4777458-v9hv8"] Oct 31 00:21:06.795452 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:06.795426 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7731d80_c8a6_4db2_a068_d89a39303720.slice/crio-5e538ffd7ffb21ba76f9110ecbcb044e1b3a91bf6122850316c088ce4945cbf6 WatchSource:0}: Error finding container 5e538ffd7ffb21ba76f9110ecbcb044e1b3a91bf6122850316c088ce4945cbf6: Status 404 returned error can't find the container with id 5e538ffd7ffb21ba76f9110ecbcb044e1b3a91bf6122850316c088ce4945cbf6 Oct 31 00:21:06.861502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:06.861464 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:07.027469 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:07.027379 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:07.660817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:07.660782 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" event={"ID":"d7731d80-c8a6-4db2-a068-d89a39303720","Type":"ContainerStarted","Data":"5e538ffd7ffb21ba76f9110ecbcb044e1b3a91bf6122850316c088ce4945cbf6"} Oct 31 00:21:07.662212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:07.662188 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" event={"ID":"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad","Type":"ContainerStarted","Data":"a9fb47d8e6ba2820735bf5bd374e72778a03c90138593ab8a86bc0570b159baf"} Oct 31 00:21:08.669471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:08.669406 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" event={"ID":"4da81e0b-257e-45dc-bbed-88ca3589a4b4","Type":"ContainerStarted","Data":"61fb82ea14e3ae010143f4080b61a6ba4e89b46a8df785b7a64627635a13462d"} Oct 31 00:21:08.671655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:08.671624 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" event={"ID":"ed2996a2-f3db-4ddf-bd16-0672659200fe","Type":"ContainerStarted","Data":"e9374191ad04a0503d01ca32331a75b391c1f015cd47f501489e79babf4794ed"} Oct 31 00:21:08.671855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:08.671834 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:08.789068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:08.789002 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-zxhr6" podStartSLOduration=2.371946745 podStartE2EDuration="5.788979294s" totalImagesPullingTime="3.417032549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:08.786543847 +0000 UTC m=+395.711812791" watchObservedRunningTime="2026-10-31 00:21:08.788979294 +0000 UTC m=+395.714248249" Oct 31 00:21:08.910969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:08.910907 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" podStartSLOduration=2.092942021 podStartE2EDuration="6.910886739s" totalImagesPullingTime="4.817944718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:08.910155739 +0000 UTC m=+395.835424692" watchObservedRunningTime="2026-10-31 00:21:08.910886739 +0000 UTC m=+395.836155692" Oct 31 00:21:09.394581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:09.394547 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:09.562370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:09.562333 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:10.106394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.106359 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:10.260722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.260684 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:10.613223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.613196 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:10.681720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.681684 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" event={"ID":"d7731d80-c8a6-4db2-a068-d89a39303720","Type":"ContainerStarted","Data":"fcac4bac59b35d148a3b6281dee387c9cc01877d314ae5183de463af493e420f"} Oct 31 00:21:10.683234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.683212 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" event={"ID":"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad","Type":"ContainerStarted","Data":"1dac51778a32418d5cf4174dad40b050aff33e6f5d6a3af30deb52f65a52462d"} Oct 31 00:21:10.683337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.683298 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:10.794888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.794835 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66d4777458-v9hv8" podStartSLOduration=2.2768727970000002 podStartE2EDuration="5.794821913s" totalImagesPullingTime="3.517949116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:10.793512374 +0000 UTC m=+397.718781326" watchObservedRunningTime="2026-10-31 00:21:10.794821913 +0000 UTC m=+397.720090865" Oct 31 00:21:10.902177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:10.902121 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" podStartSLOduration=2.14236472 podStartE2EDuration="5.902106772s" totalImagesPullingTime="3.759742052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:10.901378694 +0000 UTC m=+397.826647646" watchObservedRunningTime="2026-10-31 00:21:10.902106772 +0000 UTC m=+397.827375724" Oct 31 00:21:13.458319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:13.458248 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:14.328156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:14.328124 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:14.546304 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:14.546269 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:14.678922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:14.678893 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-nck4v" Oct 31 00:21:14.709524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:14.709496 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:15.719154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:15.719125 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:15.944144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:15.944111 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:16.860743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:16.860710 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.044759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:17.044724 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.267943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:17.267901 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.423333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:17.423290 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:17.876725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:17.876690 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:21.687853 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:21.687822 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:23.402235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:23.402197 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:23.549643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:23.549608 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:26.372709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:26.372671 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:26.373106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:26.372865 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" podUID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" containerName="limitador" containerID="cri-o://1dac51778a32418d5cf4174dad40b050aff33e6f5d6a3af30deb52f65a52462d" gracePeriod=30 Oct 31 00:21:26.744715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:26.744683 4471 generic.go:356] "Generic (PLEG): container finished" podID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" containerID="1dac51778a32418d5cf4174dad40b050aff33e6f5d6a3af30deb52f65a52462d" exitCode=0 Oct 31 00:21:26.744862 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:26.744756 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" event={"ID":"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad","Type":"ContainerDied","Data":"1dac51778a32418d5cf4174dad40b050aff33e6f5d6a3af30deb52f65a52462d"} Oct 31 00:21:27.300718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.300691 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:27.434146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.434117 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\" (UniqueName: \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\") pod \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " Oct 31 00:21:27.434513 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.434157 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\" (UniqueName: \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\") pod \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\" (UID: \"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad\") " Oct 31 00:21:27.434572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.434516 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file" pod "e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" (UID: "e9bdf63e-eadc-4fc5-b007-0004edfcb1ad"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:21:27.436269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.436226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26" pod "e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" (UID: "e9bdf63e-eadc-4fc5-b007-0004edfcb1ad"). InnerVolumeSpecName "kube-api-access-p5v26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:21:27.534721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.534691 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5v26\" (UniqueName: \"kubernetes.io/projected/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-kube-api-access-p5v26\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:27.534721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.534717 4471 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad-config-file\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:21:27.749861 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.749834 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" Oct 31 00:21:27.750013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.749858 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jkwrv" event={"ID":"e9bdf63e-eadc-4fc5-b007-0004edfcb1ad","Type":"ContainerDied","Data":"a9fb47d8e6ba2820735bf5bd374e72778a03c90138593ab8a86bc0570b159baf"} Oct 31 00:21:27.750013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.749913 4471 scope.go:122] "RemoveContainer" containerID="1dac51778a32418d5cf4174dad40b050aff33e6f5d6a3af30deb52f65a52462d" Oct 31 00:21:27.903932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.903897 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:27.949708 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:27.949637 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jkwrv"] Oct 31 00:21:29.207071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:29.207037 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" path="/var/lib/kubelet/pods/e9bdf63e-eadc-4fc5-b007-0004edfcb1ad/volumes" Oct 31 00:21:30.433929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:30.433898 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:30.584716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:30.584683 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:34.278667 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:21:34.278630 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:21:34.987958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.987921 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf"] Oct 31 00:21:34.988251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.988239 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" containerName="limitador" Oct 31 00:21:34.988291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.988251 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" containerName="limitador" Oct 31 00:21:34.988324 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.988320 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bdf63e-eadc-4fc5-b007-0004edfcb1ad" containerName="limitador" Oct 31 00:21:34.991373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.991355 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:34.997317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.997291 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80847ae4-029e-42e4-8ed2-648f67e4242e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:34.997470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.997332 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hbj\" (UniqueName: \"kubernetes.io/projected/80847ae4-029e-42e4-8ed2-648f67e4242e-kube-api-access-p2hbj\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:34.997823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.997459 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:34.997823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:34.997506 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.040188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.040161 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:21:35.040314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.040262 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:21:35.040314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.040274 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-k57ph\"" Oct 31 00:21:35.098782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.098751 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80847ae4-029e-42e4-8ed2-648f67e4242e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.098930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.098795 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hbj\" (UniqueName: \"kubernetes.io/projected/80847ae4-029e-42e4-8ed2-648f67e4242e-kube-api-access-p2hbj\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.098930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.098857 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.098930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.098892 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.099187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.099169 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.099233 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.099217 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80847ae4-029e-42e4-8ed2-648f67e4242e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.133772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.133744 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 00:21:35.140483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.140462 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80847ae4-029e-42e4-8ed2-648f67e4242e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.180444 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.180406 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf"] Oct 31 00:21:35.181749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.181727 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hbj\" (UniqueName: \"kubernetes.io/projected/80847ae4-029e-42e4-8ed2-648f67e4242e-kube-api-access-p2hbj\") pod \"kserve-module-controller-manager-7566cf5fc8-bs7tf\" (UID: \"80847ae4-029e-42e4-8ed2-648f67e4242e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.245724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.245651 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:35.302806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.302781 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:35.398356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.398321 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:35.774687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.774662 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf"] Oct 31 00:21:35.777106 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:35.777074 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80847ae4_029e_42e4_8ed2_648f67e4242e.slice/crio-a03a3cc70e87113a6648f307aa4e8a2c2c4e7b8d4119e86e2683c434efcecbad WatchSource:0}: Error finding container a03a3cc70e87113a6648f307aa4e8a2c2c4e7b8d4119e86e2683c434efcecbad: Status 404 returned error can't find the container with id a03a3cc70e87113a6648f307aa4e8a2c2c4e7b8d4119e86e2683c434efcecbad Oct 31 00:21:35.794045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:35.794013 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" event={"ID":"80847ae4-029e-42e4-8ed2-648f67e4242e","Type":"ContainerStarted","Data":"a03a3cc70e87113a6648f307aa4e8a2c2c4e7b8d4119e86e2683c434efcecbad"} Oct 31 00:21:37.727726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:37.727688 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:37.884503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:37.884471 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:39.812408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:39.812372 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" event={"ID":"80847ae4-029e-42e4-8ed2-648f67e4242e","Type":"ContainerStarted","Data":"e3d1ca675e72b81f5ba3ea160d138ab944f38a52c77aa9b19259d019f64d1a56"} Oct 31 00:21:39.812815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:39.812450 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:21:39.972325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:39.972269 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" podStartSLOduration=2.62612148 podStartE2EDuration="5.972254033s" totalImagesPullingTime="3.346132553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:39.970766673 +0000 UTC m=+426.896035626" watchObservedRunningTime="2026-10-31 00:21:39.972254033 +0000 UTC m=+426.897522985" Oct 31 00:21:42.248260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:42.248210 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:42.436061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:42.436022 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:43.016370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.016336 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9"] Oct 31 00:21:43.023867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.023841 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.072497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.072459 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dqs\" (UniqueName: \"kubernetes.io/projected/cde4498f-bec4-4e51-b49d-b5161e82d106-kube-api-access-c8dqs\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.072680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.072499 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cde4498f-bec4-4e51-b49d-b5161e82d106-data\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.075018 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.074989 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9"] Oct 31 00:21:43.116697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.116649 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:43.120201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.120178 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:21:43.120201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.120197 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v9rrl\"" Oct 31 00:21:43.120338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.120178 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 00:21:43.122878 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.122861 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:21:43.173177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.173147 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dqs\" (UniqueName: \"kubernetes.io/projected/cde4498f-bec4-4e51-b49d-b5161e82d106-kube-api-access-c8dqs\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.173320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.173182 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cde4498f-bec4-4e51-b49d-b5161e82d106-data\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.173630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.173608 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/cde4498f-bec4-4e51-b49d-b5161e82d106-data\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.234144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.234113 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dqs\" (UniqueName: \"kubernetes.io/projected/cde4498f-bec4-4e51-b49d-b5161e82d106-kube-api-access-c8dqs\") pod \"postgres-7cdbf6dc44-qq8l9\" (UID: \"cde4498f-bec4-4e51-b49d-b5161e82d106\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.335409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.335327 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:43.695286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.695256 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9"] Oct 31 00:21:43.698369 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:21:43.698341 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde4498f_bec4_4e51_b49d_b5161e82d106.slice/crio-bd72115c05fe8c402f555c6146cdc5c00da0519591cb87f7587263606d8b3f79 WatchSource:0}: Error finding container bd72115c05fe8c402f555c6146cdc5c00da0519591cb87f7587263606d8b3f79: Status 404 returned error can't find the container with id bd72115c05fe8c402f555c6146cdc5c00da0519591cb87f7587263606d8b3f79 Oct 31 00:21:43.829569 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:43.829516 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" event={"ID":"cde4498f-bec4-4e51-b49d-b5161e82d106","Type":"ContainerStarted","Data":"bd72115c05fe8c402f555c6146cdc5c00da0519591cb87f7587263606d8b3f79"} Oct 31 00:21:44.640851 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:44.640789 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:44.835480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:44.835416 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.041315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:45.041231 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.200220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:45.200185 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.452992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:45.452945 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:45.620087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:45.620053 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:51.869237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:51.869203 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" event={"ID":"cde4498f-bec4-4e51-b49d-b5161e82d106","Type":"ContainerStarted","Data":"661cb5db91aa19d67d5feaafde15ad7d58beef02479254d440628365b730e025"} Oct 31 00:21:51.869639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:51.869328 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:51.977646 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:51.977594 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" podStartSLOduration=2.709119081 podStartE2EDuration="9.977543933s" totalImagesPullingTime="7.268424852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:51.977239496 +0000 UTC m=+438.902508449" watchObservedRunningTime="2026-10-31 00:21:51.977543933 +0000 UTC m=+438.902812940" Oct 31 00:21:52.044807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:52.044774 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:52.197192 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:52.197163 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:56.846415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:56.846380 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:56.991843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:56.991807 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:57.901124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:57.901092 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qq8l9" Oct 31 00:21:59.290265 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:59.290224 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:59.435065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:21:59.435035 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:00.818414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:00.818387 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-bs7tf" Oct 31 00:22:04.279759 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:22:04.279719 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:22:05.170630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:05.170592 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.318519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:05.318473 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.468840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:05.468760 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.625222 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:05.625187 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:05.772887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:05.772809 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:10.346773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:10.346743 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:10.502278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:10.502235 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.362167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:13.362128 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:13.683638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:13.683604 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:14.289274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:14.289241 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:18.209376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:18.209344 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:18.360426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:18.360393 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:21.940015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:21.939977 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:34.278610 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:22:34.278573 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:22:40.776886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:40.776854 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:40.936376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:40.936343 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.122185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.122108 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.296641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.296604 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.458630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.458599 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.627786 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.627748 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.758911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.758827 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:22:41.785738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.785716 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:41.805488 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.805459 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:41.818338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.818316 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:22:41.863365 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.863330 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gckdg\"" Oct 31 00:22:41.868123 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.868099 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fxl\" (UniqueName: \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:41.868213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.868150 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:41.881942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.881923 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 00:22:41.968854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.968821 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fxl\" (UniqueName: \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:41.968998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.968863 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:41.971163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:41.971137 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:42.026576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:42.026494 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fxl\" (UniqueName: \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\") pod \"llmisvc-controller-manager-869d5dcc5d-ppxfg\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:42.096681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:42.096649 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:42.457760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:42.457738 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:22:42.460234 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:22:42.460204 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaee347d3_be38_4eb7_ab03_83f45566476c.slice/crio-c4611ae6e40f8bc2be48b413f8890ee2f748a879c65d2245107e38ee5ab5f77f WatchSource:0}: Error finding container c4611ae6e40f8bc2be48b413f8890ee2f748a879c65d2245107e38ee5ab5f77f: Status 404 returned error can't find the container with id c4611ae6e40f8bc2be48b413f8890ee2f748a879c65d2245107e38ee5ab5f77f Oct 31 00:22:42.970204 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:42.970172 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.089201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:43.089162 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" event={"ID":"aee347d3-be38-4eb7-ab03-83f45566476c","Type":"ContainerStarted","Data":"c4611ae6e40f8bc2be48b413f8890ee2f748a879c65d2245107e38ee5ab5f77f"} Oct 31 00:22:43.152208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:43.152172 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.361501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:43.361397 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.508704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:43.508670 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:43.656656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:43.656622 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:44.264843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.264809 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:22:44.268106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.268089 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.363618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.363566 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4nxh\"" Oct 31 00:22:44.363618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.363572 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 00:22:44.363618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.363592 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:22:44.365178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.365155 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:22:44.391813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.391789 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.391945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.391851 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqbc\" (UniqueName: \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.391945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.391908 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.392024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.391962 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.493211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.493181 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqbc\" (UniqueName: \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.493211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.493218 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.493428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.493245 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.493428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.493313 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.494038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.494002 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.495946 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.495919 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.496075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.495968 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.550872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.550801 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqbc\" (UniqueName: \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\") pod \"maas-api-6d669c7fcd-bkmb2\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.580779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.580748 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:44.947676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:44.947652 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:22:44.950438 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:22:44.950414 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dc31ee_e5df_4421_8711_e90a26260bd3.slice/crio-329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f WatchSource:0}: Error finding container 329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f: Status 404 returned error can't find the container with id 329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f Oct 31 00:22:45.099357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:45.099277 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" event={"ID":"aee347d3-be38-4eb7-ab03-83f45566476c","Type":"ContainerStarted","Data":"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de"} Oct 31 00:22:45.099514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:45.099383 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:22:45.100460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:45.100437 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" event={"ID":"10dc31ee-e5df-4421-8711-e90a26260bd3","Type":"ContainerStarted","Data":"329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f"} Oct 31 00:22:45.211019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:45.210972 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" podStartSLOduration=1.844097409 podStartE2EDuration="4.210958681s" totalImagesPullingTime="2.366861272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:45.209624968 +0000 UTC m=+492.134893913" watchObservedRunningTime="2026-10-31 00:22:45.210958681 +0000 UTC m=+492.136227675" Oct 31 00:22:47.966805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:47.966770 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.176880 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.176843 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.323237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.323210 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.473451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.473417 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.623157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.623081 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.825029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.825002 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:48.969717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:48.969683 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.117040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:49.117003 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:49.122426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:49.122397 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" event={"ID":"10dc31ee-e5df-4421-8711-e90a26260bd3","Type":"ContainerStarted","Data":"0c08d2eec32b4b117c717905ab5e4d6942af97914ef856f231ac0c39d01a9b85"} Oct 31 00:22:49.122589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:49.122454 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:49.233781 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:49.233705 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" podStartSLOduration=1.9160423039999999 podStartE2EDuration="5.233693191s" totalImagesPullingTime="3.317650887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:49.232749179 +0000 UTC m=+496.158018131" watchObservedRunningTime="2026-10-31 00:22:49.233693191 +0000 UTC m=+496.158962137" Oct 31 00:22:55.131791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:55.131756 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:22:55.678205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:55.678176 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:55.944345 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:22:55.944282 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:04.278721 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:23:04.278685 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:23:07.971710 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:07.971678 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:23:07.972425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:07.971877 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" podUID="10dc31ee-e5df-4421-8711-e90a26260bd3" containerName="maas-api" containerID="cri-o://0c08d2eec32b4b117c717905ab5e4d6942af97914ef856f231ac0c39d01a9b85" gracePeriod=30 Oct 31 00:23:08.209478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.209448 4471 generic.go:356] "Generic (PLEG): container finished" podID="10dc31ee-e5df-4421-8711-e90a26260bd3" containerID="0c08d2eec32b4b117c717905ab5e4d6942af97914ef856f231ac0c39d01a9b85" exitCode=0 Oct 31 00:23:08.209622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.209521 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" event={"ID":"10dc31ee-e5df-4421-8711-e90a26260bd3","Type":"ContainerDied","Data":"0c08d2eec32b4b117c717905ab5e4d6942af97914ef856f231ac0c39d01a9b85"} Oct 31 00:23:08.209622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.209592 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" event={"ID":"10dc31ee-e5df-4421-8711-e90a26260bd3","Type":"ContainerDied","Data":"329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f"} Oct 31 00:23:08.209622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.209609 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="329ab7bf80aece058d2e8d7aeb3d9709f7bcc23fe7f560f912db42c966e4a78f" Oct 31 00:23:08.223097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.223038 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:23:08.386695 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.386661 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\") pod \"10dc31ee-e5df-4421-8711-e90a26260bd3\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " Oct 31 00:23:08.386854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.386755 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\") pod \"10dc31ee-e5df-4421-8711-e90a26260bd3\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " Oct 31 00:23:08.386854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.386779 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\") pod \"10dc31ee-e5df-4421-8711-e90a26260bd3\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " Oct 31 00:23:08.386854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.386820 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\" (UniqueName: \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\") pod \"10dc31ee-e5df-4421-8711-e90a26260bd3\" (UID: \"10dc31ee-e5df-4421-8711-e90a26260bd3\") " Oct 31 00:23:08.387190 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.387163 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca" pod "10dc31ee-e5df-4421-8711-e90a26260bd3" (UID: "10dc31ee-e5df-4421-8711-e90a26260bd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:08.388945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.388923 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls" pod "10dc31ee-e5df-4421-8711-e90a26260bd3" (UID: "10dc31ee-e5df-4421-8711-e90a26260bd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:08.389043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.388972 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs" pod "10dc31ee-e5df-4421-8711-e90a26260bd3" (UID: "10dc31ee-e5df-4421-8711-e90a26260bd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:08.389043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.389036 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc" pod "10dc31ee-e5df-4421-8711-e90a26260bd3" (UID: "10dc31ee-e5df-4421-8711-e90a26260bd3"). InnerVolumeSpecName "kube-api-access-wpqbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:08.487769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.487682 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpqbc\" (UniqueName: \"kubernetes.io/projected/10dc31ee-e5df-4421-8711-e90a26260bd3-kube-api-access-wpqbc\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:08.487769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.487714 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:08.487769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.487729 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10dc31ee-e5df-4421-8711-e90a26260bd3-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:08.487769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:08.487741 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10dc31ee-e5df-4421-8711-e90a26260bd3-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:09.213948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:09.213809 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2" Oct 31 00:23:09.391635 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:09.391601 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:23:09.437457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:09.437424 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-bkmb2"] Oct 31 00:23:11.211694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:11.211658 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dc31ee-e5df-4421-8711-e90a26260bd3" path="/var/lib/kubelet/pods/10dc31ee-e5df-4421-8711-e90a26260bd3/volumes" Oct 31 00:23:16.107002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:16.106966 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:23:18.558632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:18.558595 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:19.961177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:19.961147 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:20.111938 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:20.111907 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:20.256885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:23:20.256810 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:34.278667 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:23:34.278623 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:24:01.528431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.528395 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mfmng"] Oct 31 00:24:01.529052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.528896 4471 cpu_manager.go:414] "containerMap: removing container" podUID="10dc31ee-e5df-4421-8711-e90a26260bd3" containerName="maas-api" Oct 31 00:24:01.529052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.528914 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dc31ee-e5df-4421-8711-e90a26260bd3" containerName="maas-api" Oct 31 00:24:01.529052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.529026 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dc31ee-e5df-4421-8711-e90a26260bd3" containerName="maas-api" Oct 31 00:24:01.531496 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.531476 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.594717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.594603 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mfmng"] Oct 31 00:24:01.641093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.641055 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b270816d-d116-40b5-9d03-955a269225bd-cert\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.641093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.641096 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2d2\" (UniqueName: \"kubernetes.io/projected/b270816d-d116-40b5-9d03-955a269225bd-kube-api-access-hj2d2\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.741673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.741637 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b270816d-d116-40b5-9d03-955a269225bd-cert\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.741841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.741680 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2d2\" (UniqueName: \"kubernetes.io/projected/b270816d-d116-40b5-9d03-955a269225bd-kube-api-access-hj2d2\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.748097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.748062 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b270816d-d116-40b5-9d03-955a269225bd-cert\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.800037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.799963 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2d2\" (UniqueName: \"kubernetes.io/projected/b270816d-d116-40b5-9d03-955a269225bd-kube-api-access-hj2d2\") pod \"llmisvc-controller-manager-6466c554d8-mfmng\" (UID: \"b270816d-d116-40b5-9d03-955a269225bd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:01.842870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:01.842839 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:02.205920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:02.205893 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mfmng"] Oct 31 00:24:02.208594 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:24:02.208564 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb270816d_d116_40b5_9d03_955a269225bd.slice/crio-014b998d70a73d34fb12398d5e6445862451dfc54d0691dd2141986f0afc5b0b WatchSource:0}: Error finding container 014b998d70a73d34fb12398d5e6445862451dfc54d0691dd2141986f0afc5b0b: Status 404 returned error can't find the container with id 014b998d70a73d34fb12398d5e6445862451dfc54d0691dd2141986f0afc5b0b Oct 31 00:24:02.444544 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:02.444498 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" event={"ID":"b270816d-d116-40b5-9d03-955a269225bd","Type":"ContainerStarted","Data":"014b998d70a73d34fb12398d5e6445862451dfc54d0691dd2141986f0afc5b0b"} Oct 31 00:24:03.449630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:03.449596 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" event={"ID":"b270816d-d116-40b5-9d03-955a269225bd","Type":"ContainerStarted","Data":"dcc7e4a04f4f59620e191e3ada9926ff38c847f490461f0fc9494626e01d7eeb"} Oct 31 00:24:03.450080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:03.449669 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:03.559730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:03.559511 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" podStartSLOduration=1.681226506 podStartE2EDuration="2.559494235s" totalImagesPullingTime="878.267729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:03.558826135 +0000 UTC m=+570.484095088" watchObservedRunningTime="2026-10-31 00:24:03.559494235 +0000 UTC m=+570.484763188" Oct 31 00:24:04.279373 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:24:04.279335 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:24:15.380713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:15.380637 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:15.524449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:15.524409 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:18.918749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:18.918705 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:28.412180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:28.412148 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:34.278462 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:24:34.278417 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:24:34.455448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.455417 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mfmng" Oct 31 00:24:34.612860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.612782 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:24:34.613035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.612979 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" podUID="aee347d3-be38-4eb7-ab03-83f45566476c" containerName="manager" containerID="cri-o://a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de" gracePeriod=30 Oct 31 00:24:34.846285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.846259 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:24:34.911719 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.911690 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\" (UniqueName: \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\") pod \"aee347d3-be38-4eb7-ab03-83f45566476c\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " Oct 31 00:24:34.911875 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.911756 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\" (UniqueName: \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\") pod \"aee347d3-be38-4eb7-ab03-83f45566476c\" (UID: \"aee347d3-be38-4eb7-ab03-83f45566476c\") " Oct 31 00:24:34.913836 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.913813 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl" pod "aee347d3-be38-4eb7-ab03-83f45566476c" (UID: "aee347d3-be38-4eb7-ab03-83f45566476c"). InnerVolumeSpecName "kube-api-access-s9fxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:34.913954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:34.913863 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert" pod "aee347d3-be38-4eb7-ab03-83f45566476c" (UID: "aee347d3-be38-4eb7-ab03-83f45566476c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:24:35.013298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.013266 4471 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aee347d3-be38-4eb7-ab03-83f45566476c-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:35.013298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.013293 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9fxl\" (UniqueName: \"kubernetes.io/projected/aee347d3-be38-4eb7-ab03-83f45566476c-kube-api-access-s9fxl\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:35.446676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.446641 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:35.575327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.575243 4471 generic.go:356] "Generic (PLEG): container finished" podID="aee347d3-be38-4eb7-ab03-83f45566476c" containerID="a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de" exitCode=0 Oct 31 00:24:35.575327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.575294 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" event={"ID":"aee347d3-be38-4eb7-ab03-83f45566476c","Type":"ContainerDied","Data":"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de"} Oct 31 00:24:35.575327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.575303 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" Oct 31 00:24:35.575604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.575335 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg" event={"ID":"aee347d3-be38-4eb7-ab03-83f45566476c","Type":"ContainerDied","Data":"c4611ae6e40f8bc2be48b413f8890ee2f748a879c65d2245107e38ee5ab5f77f"} Oct 31 00:24:35.575604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.575360 4471 scope.go:122] "RemoveContainer" containerID="a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de" Oct 31 00:24:35.583306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.583287 4471 scope.go:122] "RemoveContainer" containerID="a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de" Oct 31 00:24:35.583592 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:24:35.583572 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de\": container with ID starting with a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de not found: ID does not exist" containerID="a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de" Oct 31 00:24:35.583642 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.583601 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de"} err="failed to get container status \"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de\": rpc error: code = NotFound desc = could not find container \"a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de\": container with ID starting with a5bd25a9059839db9a3340cdbfba72097e1096375ce2a893db2079998108d9de not found: ID does not exist" Oct 31 00:24:35.601465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.601445 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:35.742215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.742180 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:24:35.788661 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:35.788629 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ppxfg"] Oct 31 00:24:37.206609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:37.206578 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee347d3-be38-4eb7-ab03-83f45566476c" path="/var/lib/kubelet/pods/aee347d3-be38-4eb7-ab03-83f45566476c/volumes" Oct 31 00:24:54.394343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:24:54.394300 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:00.947969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:00.947935 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:03.840191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:03.840159 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:03.984063 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:03.984026 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:04.279768 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:25:04.279731 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:25:04.578166 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:04.578082 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:07.461854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:07.461822 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:11.509605 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:11.509566 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:16.297678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:16.297646 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:19.258150 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:19.258110 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:20.187189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:20.187152 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:23.548817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:23.548783 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:31.507675 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.507636 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:25:31.508156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.508056 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aee347d3-be38-4eb7-ab03-83f45566476c" containerName="manager" Oct 31 00:25:31.508156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.508074 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee347d3-be38-4eb7-ab03-83f45566476c" containerName="manager" Oct 31 00:25:31.508156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.508134 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee347d3-be38-4eb7-ab03-83f45566476c" containerName="manager" Oct 31 00:25:31.511207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.511188 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.561014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.560984 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fxh6q\"" Oct 31 00:25:31.605757 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605718 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.605955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605772 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.605955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605852 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.605955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605882 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.605955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605908 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.606219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.605968 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmtv7\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.606219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.606086 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.606219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.606138 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.606219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.606168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.607832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.607809 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:25:31.707024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.706986 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707032 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707058 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707166 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmtv7\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707207 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707238 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707278 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707366 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707581 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707640 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707757 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.707792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707773 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.708010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.707994 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.709610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.709591 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.709697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.709667 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.764821 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.764743 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.764975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.764838 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmtv7\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\") pod \"maas-default-gateway-openshift-default-6886b96f9-c2j7f\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:31.823513 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:31.823468 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:32.243415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.243338 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:25:32.246619 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:25:32.246588 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa66a9f_1f63_4b29_8ce1_7ae32d47a96e.slice/crio-4a8af970a820cae79c7b2ec3b725a1a81e880d11314ea190ca76c8a868923acb WatchSource:0}: Error finding container 4a8af970a820cae79c7b2ec3b725a1a81e880d11314ea190ca76c8a868923acb: Status 404 returned error can't find the container with id 4a8af970a820cae79c7b2ec3b725a1a81e880d11314ea190ca76c8a868923acb Oct 31 00:25:32.248812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.248785 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:25:32.248877 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.248845 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:25:32.248912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.248874 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:25:32.822418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.822381 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" event={"ID":"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e","Type":"ContainerStarted","Data":"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9"} Oct 31 00:25:32.822922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.822431 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" event={"ID":"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e","Type":"ContainerStarted","Data":"4a8af970a820cae79c7b2ec3b725a1a81e880d11314ea190ca76c8a868923acb"} Oct 31 00:25:32.823638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.823623 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:32.938387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:32.938335 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podStartSLOduration=1.93831922 podStartE2EDuration="1.93831922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:32.936174338 +0000 UTC m=+659.861443294" watchObservedRunningTime="2026-10-31 00:25:32.93831922 +0000 UTC m=+659.863588169" Oct 31 00:25:33.825267 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:33.825224 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.825713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:33.825277 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:34.230923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:34.230888 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:34.304792 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:25:34.304756 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:25:34.824268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:34.824234 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:34.824465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:34.824300 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:35.824012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:35.823966 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:35.824501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:35.824030 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:36.824845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:36.824804 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:36.825271 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:36.824862 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:37.824637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:37.824596 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:37.824825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:37.824656 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:38.824635 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:38.824591 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 31 00:25:38.825236 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:38.824656 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 31 00:25:40.234800 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:40.234772 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:40.235225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:40.235140 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:40.236032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:40.236013 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:25:49.188022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:49.187936 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:49.448753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:49.448662 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:59.728789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:59.728757 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:59.898308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:25:59.898270 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:04.048208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:04.048168 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:04.282403 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:26:04.282365 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:26:19.588211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:19.588172 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:22.737737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:22.737690 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:23.268311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:23.268275 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:24.948122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:24.948085 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:25.097305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:25.097278 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:26.570891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:26.570859 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:26.739064 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:26.739027 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:34.279202 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:26:34.279162 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:26:51.961138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:51.961102 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.110864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:52.110830 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.619045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:52.619012 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.792580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:52.792543 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:52.944798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:52.944753 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.168298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.168267 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:26:53.176698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.176450 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.233199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.232867 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-1e5e75-openshift-default-dockercfg-gw4mk\"" Oct 31 00:26:53.239139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239101 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239163 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239284 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239429 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239320 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239429 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239394 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239429 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239421 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239438 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfj9z\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239466 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239723 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239635 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.239923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.239902 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.268763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.268717 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:26:53.340165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340129 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfj9z\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340187 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340231 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340275 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340317 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340342 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340404 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340431 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340768 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.340915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340892 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.341322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.340954 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.341322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.341261 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.341632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.341604 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.343799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.343776 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.344097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.344079 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.387257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.387211 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.403130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.403095 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.403286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.403209 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfj9z\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\") pod \"e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.491038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.490951 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:53.557098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.556982 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.707592 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.707555 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.863138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.863046 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:53.924596 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:26:53.924556 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71de909a_5700_4360_8d6d_6551809d6a76.slice/crio-becf8894e4e31724b16a6865e8eb98fd351f33c354398e753a788810b76b51f0 WatchSource:0}: Error finding container becf8894e4e31724b16a6865e8eb98fd351f33c354398e753a788810b76b51f0: Status 404 returned error can't find the container with id becf8894e4e31724b16a6865e8eb98fd351f33c354398e753a788810b76b51f0 Oct 31 00:26:53.924937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.924916 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:26:53.927180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.927139 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:53.927265 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.927229 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:53.927305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:53.927268 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:54.012055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.011446 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.175726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.175680 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerStarted","Data":"bb5a6a1d87912dc734790d5aea16ad40700a828ac87fa7184230c931bb783a96"} Oct 31 00:26:54.176124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.175742 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerStarted","Data":"becf8894e4e31724b16a6865e8eb98fd351f33c354398e753a788810b76b51f0"} Oct 31 00:26:54.253377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.253345 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:26:54.256746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.256727 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.307502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.307436 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podStartSLOduration=1.307416576 podStartE2EDuration="1.307416576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:54.305445109 +0000 UTC m=+741.230714087" watchObservedRunningTime="2026-10-31 00:26:54.307416576 +0000 UTC m=+741.232685530" Oct 31 00:26:54.310289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.310269 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-101b9e-openshift-default-dockercfg-8l66m\"" Oct 31 00:26:54.319086 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.319054 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:26:54.336856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.336797 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.348257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348224 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348259 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348281 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348356 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348488 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348517 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348584 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348615 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtsp\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.348816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.348644 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449022 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449059 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449079 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449107 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449125 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449142 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449172 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449207 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtsp\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449237 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449560 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449593 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449693 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.449871 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449856 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.450009 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.449988 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.451507 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.451480 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.451665 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.451628 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.485161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.485112 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.492116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.492086 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:54.497509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.497486 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:54.509988 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.509967 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.510272 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.510252 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtsp\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.570611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.570583 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:54.648196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.646594 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:54.805927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:54.805833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.004758 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:26:55.004726 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959c2dea_6fbd_40af_8f9e_063113a88298.slice/crio-85b052b6a9605ec9a0f8c52c963b99dfd6590047f125e4ac5c64b72e5baa752f WatchSource:0}: Error finding container 85b052b6a9605ec9a0f8c52c963b99dfd6590047f125e4ac5c64b72e5baa752f: Status 404 returned error can't find the container with id 85b052b6a9605ec9a0f8c52c963b99dfd6590047f125e4ac5c64b72e5baa752f Oct 31 00:26:55.005655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.005630 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:26:55.006974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.006946 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:55.007043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.007010 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:55.007078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.007039 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:26:55.048702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.048672 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:55.184549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.183828 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerStarted","Data":"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69"} Oct 31 00:26:55.184549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.183882 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerStarted","Data":"85b052b6a9605ec9a0f8c52c963b99dfd6590047f125e4ac5c64b72e5baa752f"} Oct 31 00:26:55.184549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.183912 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:55.185014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.184570 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:26:55.305468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.305417 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podStartSLOduration=1.305404596 podStartE2EDuration="1.305404596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:55.303890334 +0000 UTC m=+742.229159287" watchObservedRunningTime="2026-10-31 00:26:55.305404596 +0000 UTC m=+742.230673548" Oct 31 00:26:55.571182 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:55.571132 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:56.575775 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:56.575746 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:56.849781 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:56.849677 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.072815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:57.072745 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.191706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:57.191677 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:57.192522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:57.192500 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:26:57.231430 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:57.231402 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:57.966864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:57.966833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.160718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:58.160686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.605361 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:58.605330 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:58.838546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:58.838500 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.239394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:59.239363 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.424175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:59.424146 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.640038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:59.639949 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:59.854002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:26:59.853973 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.430953 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:01.430916 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.591351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:01.591316 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.743177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:01.743105 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:01.914197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:01.914154 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:02.086425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:02.086341 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:02.273216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:02.273177 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:02.449433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:02.449402 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:03.928006 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:03.927969 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.093583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:04.093550 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.248031 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:04.247951 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:04.280467 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:04.280433 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:27:06.570822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:06.570781 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.734247 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:06.734211 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.918902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:06.918867 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:06.962136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:06.962106 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:27:06.965671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:06.965653 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.020173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.020135 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-1e5e75\"" Oct 31 00:27:07.021950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.021918 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:27:07.058316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.058280 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59bg\" (UniqueName: \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.058449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.058431 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.070030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.070002 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-1e5e75-dockercfg-2p47m\"" Oct 31 00:27:07.126136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.126103 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.159858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.159821 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.160030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.159882 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59bg\" (UniqueName: \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.160697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.160677 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.223208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.223138 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59bg\" (UniqueName: \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\") pod \"payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.279033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.278993 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:07.358706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.342920 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:07.662679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.662650 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:27:07.663113 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:07.662711 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763d31cb_8503_4af7_a2b0_fe264ed5604b.slice/crio-e58d1e8a552813aa8d395fcaa9b702b579cd4fe7f84f5cdff037b4e4ed615546 WatchSource:0}: Error finding container e58d1e8a552813aa8d395fcaa9b702b579cd4fe7f84f5cdff037b4e4ed615546: Status 404 returned error can't find the container with id e58d1e8a552813aa8d395fcaa9b702b579cd4fe7f84f5cdff037b4e4ed615546 Oct 31 00:27:07.664801 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:07.664783 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:27:08.240218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:08.240185 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" event={"ID":"763d31cb-8503-4af7-a2b0-fe264ed5604b","Type":"ContainerStarted","Data":"e58d1e8a552813aa8d395fcaa9b702b579cd4fe7f84f5cdff037b4e4ed615546"} Oct 31 00:27:09.011580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.011544 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.184293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.184258 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.234438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.234401 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:27:09.238503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.238479 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.279356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.279280 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.279555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.279390 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.279555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.279452 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.279672 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.279545 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t96\" (UniqueName: \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.297957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.297735 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:27:09.297957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.297885 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a6f4a5-serving-cert\"" Oct 31 00:27:09.345428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.345388 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:27:09.345428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.345406 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4nxh\"" Oct 31 00:27:09.380589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.380519 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.380784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.380648 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.380784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.380672 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.380784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.380715 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t96\" (UniqueName: \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.380934 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:09.380814 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-serving-cert: secret "maas-api-e2e-worker-w6-a6f4a5-serving-cert" not found Oct 31 00:27:09.380934 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:09.380901 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls podName:8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:09.880877209 +0000 UTC m=+756.806146154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls") pod "maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" (UID: "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7") : secret "maas-api-e2e-worker-w6-a6f4a5-serving-cert" not found Oct 31 00:27:09.381782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.381757 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.383454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.383430 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.452886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.452830 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t96\" (UniqueName: \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.582359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.582282 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.769895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.769865 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:09.885522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.885435 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:09.888247 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:09.888223 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:10.155652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:10.155613 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:10.539584 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:10.539552 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8125c08e_1e9b_4ae8_8e73_19e1f44ca3f7.slice/crio-0a136c84856c7b99808d2cfdd20ef008a96d7147e6465cee48e8138d8543d25a WatchSource:0}: Error finding container 0a136c84856c7b99808d2cfdd20ef008a96d7147e6465cee48e8138d8543d25a: Status 404 returned error can't find the container with id 0a136c84856c7b99808d2cfdd20ef008a96d7147e6465cee48e8138d8543d25a Oct 31 00:27:10.540096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:10.540076 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:27:11.257555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:11.257493 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" event={"ID":"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7","Type":"ContainerStarted","Data":"0a136c84856c7b99808d2cfdd20ef008a96d7147e6465cee48e8138d8543d25a"} Oct 31 00:27:11.259028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:11.259003 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" event={"ID":"763d31cb-8503-4af7-a2b0-fe264ed5604b","Type":"ContainerStarted","Data":"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02"} Oct 31 00:27:11.259175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:11.259056 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:11.375132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:11.375072 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" podStartSLOduration=2.52202194 podStartE2EDuration="5.37505345s" totalImagesPullingTime="2.85303151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:11.374512511 +0000 UTC m=+758.299781464" watchObservedRunningTime="2026-10-31 00:27:11.37505345 +0000 UTC m=+758.300322401" Oct 31 00:27:12.265292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:12.265256 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" event={"ID":"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7","Type":"ContainerStarted","Data":"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf"} Oct 31 00:27:12.265702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:12.265563 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:12.385942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:12.385893 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" podStartSLOduration=2.486151679 podStartE2EDuration="3.385878407s" totalImagesPullingTime="899.726728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:12.384998198 +0000 UTC m=+759.310267150" watchObservedRunningTime="2026-10-31 00:27:12.385878407 +0000 UTC m=+759.311147359" Oct 31 00:27:13.782980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:13.782902 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:13.927849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:13.927806 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:14.333381 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.333350 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:27:14.336864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.336847 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.431685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.431644 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.431860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.431705 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l94f7\" (UniqueName: \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.441728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.441705 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-982ae3-dockercfg-jxrpp\"" Oct 31 00:27:14.441827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.441746 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-982ae3\"" Oct 31 00:27:14.443485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.443464 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:27:14.533036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.532998 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l94f7\" (UniqueName: \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.533183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.533093 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.533754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.533736 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.594384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.594289 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l94f7\" (UniqueName: \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\") pod \"payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.649353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.649312 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:14.908005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:14.907972 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:15.040236 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:15.040205 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f75838b_8bc3_4a19_9584_ddb728537e41.slice/crio-2efc31a2409dd3c87562e1a635ee4242914b0de7edb1713b625c4bb1200ba426 WatchSource:0}: Error finding container 2efc31a2409dd3c87562e1a635ee4242914b0de7edb1713b625c4bb1200ba426: Status 404 returned error can't find the container with id 2efc31a2409dd3c87562e1a635ee4242914b0de7edb1713b625c4bb1200ba426 Oct 31 00:27:15.041167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.041142 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:27:15.115281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.115248 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:15.281201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.281071 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:15.294910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.294872 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" event={"ID":"4f75838b-8bc3-4a19-9584-ddb728537e41","Type":"ContainerStarted","Data":"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3"} Oct 31 00:27:15.295126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.294923 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" event={"ID":"4f75838b-8bc3-4a19-9584-ddb728537e41","Type":"ContainerStarted","Data":"2efc31a2409dd3c87562e1a635ee4242914b0de7edb1713b625c4bb1200ba426"} Oct 31 00:27:15.295126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.295010 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:15.412731 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:15.412678 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" podStartSLOduration=1.4126651 podStartE2EDuration="1.4126651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:15.41110204 +0000 UTC m=+762.336370993" watchObservedRunningTime="2026-10-31 00:27:15.4126651 +0000 UTC m=+762.337934049" Oct 31 00:27:16.287993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:16.287962 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:16.448356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:16.448322 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:18.290813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:18.290781 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:27:19.914378 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:19.914341 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:20.573462 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:20.573420 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:20.933981 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:20.933925 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:21.103446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.103415 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:21.326037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.325950 4471 generic.go:356] "Generic (PLEG): container finished" podID="71de909a-5700-4360-8d6d-6551809d6a76" containerID="bb5a6a1d87912dc734790d5aea16ad40700a828ac87fa7184230c931bb783a96" exitCode=0 Oct 31 00:27:21.326037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.326007 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerDied","Data":"bb5a6a1d87912dc734790d5aea16ad40700a828ac87fa7184230c931bb783a96"} Oct 31 00:27:21.326395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.326379 4471 scope.go:122] "RemoveContainer" containerID="bb5a6a1d87912dc734790d5aea16ad40700a828ac87fa7184230c931bb783a96" Oct 31 00:27:21.327643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.327616 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:21.327715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.327702 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:21.327751 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:21.327735 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:22.282620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:22.282590 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:27:22.332090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:22.332057 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerStarted","Data":"4d244dd4b98699a46a3f91f8ca1e537fa50ba72f4a095e1855a556df5a5f8bed"} Oct 31 00:27:22.491986 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:22.491946 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:27:22.493066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:22.493036 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:27:22.493191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:22.493099 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:27:23.350476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.350387 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:23.495937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.495902 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:27:23.496152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.495967 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:27:23.526089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.526055 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:23.983783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.983732 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:27:23.989049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:23.989026 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.045239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.045201 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.045432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.045267 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cb4\" (UniqueName: \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.048724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.048697 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:27:24.095908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.095862 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-101b9e\"" Oct 31 00:27:24.096095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.095957 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-101b9e-dockercfg-mmz4v\"" Oct 31 00:27:24.146291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.146250 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.146503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.146304 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cb4\" (UniqueName: \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.147048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.147021 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.206982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.206945 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cb4\" (UniqueName: \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\") pod \"payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.302083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.301988 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:24.492160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.492117 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:27:24.492677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.492196 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:27:24.681865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:24.681645 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:27:24.686172 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:24.686140 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa043e5c_378c_43dd_997d_a3c7f0abcd72.slice/crio-950e99ab90e03861fa7758fd7041750415294f691d056f0a7b5f28de39df5c73 WatchSource:0}: Error finding container 950e99ab90e03861fa7758fd7041750415294f691d056f0a7b5f28de39df5c73: Status 404 returned error can't find the container with id 950e99ab90e03861fa7758fd7041750415294f691d056f0a7b5f28de39df5c73 Oct 31 00:27:25.184408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.184365 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:27:25.347633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.347598 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" event={"ID":"aa043e5c-378c-43dd-997d-a3c7f0abcd72","Type":"ContainerStarted","Data":"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398"} Oct 31 00:27:25.347807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.347657 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" event={"ID":"aa043e5c-378c-43dd-997d-a3c7f0abcd72","Type":"ContainerStarted","Data":"950e99ab90e03861fa7758fd7041750415294f691d056f0a7b5f28de39df5c73"} Oct 31 00:27:25.347807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.347737 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:25.478612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.478031 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" podStartSLOduration=2.478013087 podStartE2EDuration="2.478013087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:25.476924684 +0000 UTC m=+772.402193641" watchObservedRunningTime="2026-10-31 00:27:25.478013087 +0000 UTC m=+772.403282042" Oct 31 00:27:25.492147 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.492106 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:27:25.492320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.492175 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:27:25.982864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:25.982829 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:26.146354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:26.146324 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:26.299966 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:26.299881 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:26.304341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:26.304316 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:27:26.492252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:26.492217 4471 patch_prober.go:29] interesting pod/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 31 00:27:26.492460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:26.492276 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 31 00:27:27.495467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:27.495429 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:27:27.496443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:27.496424 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:27:28.997764 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:28.997732 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:29.145361 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:29.145326 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:31.004115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.004080 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 00:27:31.010146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.010126 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.113061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.113026 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-da9d07-serving-cert\"" Oct 31 00:27:31.114772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.114743 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 00:27:31.130612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.130579 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.130612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.130613 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbb2l\" (UniqueName: \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.130824 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.130727 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.130824 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.130746 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.209817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.209783 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:31.231913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.231883 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.231913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.231918 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.232239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.231956 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.232239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.231975 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbb2l\" (UniqueName: \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.232631 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.232607 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.234326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.234306 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.234425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.234390 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.295571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.295472 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbb2l\" (UniqueName: \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\") pod \"maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.323249 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.323214 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:31.371340 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.371283 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:31.514967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.514936 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:31.695805 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:31.695774 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4345d42b_60f5_4975_945a_68a767f65aa3.slice/crio-4c89b8237c9b25abf72e07ab319385c5683a792bc151541a7fdcefc6971f173b WatchSource:0}: Error finding container 4c89b8237c9b25abf72e07ab319385c5683a792bc151541a7fdcefc6971f173b: Status 404 returned error can't find the container with id 4c89b8237c9b25abf72e07ab319385c5683a792bc151541a7fdcefc6971f173b Oct 31 00:27:31.697084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:31.697061 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 00:27:32.381740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:32.381703 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" event={"ID":"4345d42b-60f5-4975-945a-68a767f65aa3","Type":"ContainerStarted","Data":"4c89b8237c9b25abf72e07ab319385c5683a792bc151541a7fdcefc6971f173b"} Oct 31 00:27:33.387880 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:33.387841 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" event={"ID":"4345d42b-60f5-4975-945a-68a767f65aa3","Type":"ContainerStarted","Data":"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f"} Oct 31 00:27:33.388309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:33.387934 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:33.505571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:33.505493 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" podStartSLOduration=2.641146289 podStartE2EDuration="3.505479993s" totalImagesPullingTime="864.333704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:33.504901345 +0000 UTC m=+780.430170316" watchObservedRunningTime="2026-10-31 00:27:33.505479993 +0000 UTC m=+780.430748946" Oct 31 00:27:34.280168 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:34.280132 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:27:36.355283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.355250 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:27:36.460160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.460128 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:27:36.464801 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.464770 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.515429 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.515392 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:27:36.517135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.517107 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-101b9e-serving-cert\"" Oct 31 00:27:36.580714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.580670 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.580714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.580711 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.580987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.580873 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.580987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.580912 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmsf\" (UniqueName: \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.658958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.658926 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:36.681692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.681652 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.681692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.681695 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmsf\" (UniqueName: \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.681899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.681794 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.681899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.681831 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.682006 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:36.681974 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-serving-cert: secret "maas-api-e2e-worker-w0-101b9e-serving-cert" not found Oct 31 00:27:36.682057 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:27:36.682048 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls podName:245d7987-2e9d-4aa0-b1c0-1640de7130e9 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:37.182025612 +0000 UTC m=+784.107294562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls") pod "maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" (UID: "245d7987-2e9d-4aa0-b1c0-1640de7130e9") : secret "maas-api-e2e-worker-w0-101b9e-serving-cert" not found Oct 31 00:27:36.682448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.682426 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.684229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.684208 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.745137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.745107 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmsf\" (UniqueName: \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:36.843895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.843855 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:36.996004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:36.995923 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:37.185391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:37.185354 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:37.187787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:37.187757 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:37.380060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:37.379978 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:37.754324 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:27:37.754282 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245d7987_2e9d_4aa0_b1c0_1640de7130e9.slice/crio-f625192d2436be3404d6a5fe8fc377a24018feb12ab538fcd4aadcf8722fca98 WatchSource:0}: Error finding container f625192d2436be3404d6a5fe8fc377a24018feb12ab538fcd4aadcf8722fca98: Status 404 returned error can't find the container with id f625192d2436be3404d6a5fe8fc377a24018feb12ab538fcd4aadcf8722fca98 Oct 31 00:27:37.756178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:37.756152 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:27:38.412753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:38.412718 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" event={"ID":"245d7987-2e9d-4aa0-b1c0-1640de7130e9","Type":"ContainerStarted","Data":"f625192d2436be3404d6a5fe8fc377a24018feb12ab538fcd4aadcf8722fca98"} Oct 31 00:27:39.399071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:39.399039 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 00:27:39.420055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:39.420020 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" event={"ID":"245d7987-2e9d-4aa0-b1c0-1640de7130e9","Type":"ContainerStarted","Data":"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35"} Oct 31 00:27:39.420567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:39.420077 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:39.642905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:39.642853 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" podStartSLOduration=2.754038933 podStartE2EDuration="3.642838119s" totalImagesPullingTime="888.799186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:39.642695825 +0000 UTC m=+786.567964789" watchObservedRunningTime="2026-10-31 00:27:39.642838119 +0000 UTC m=+786.568107071" Oct 31 00:27:42.354888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:42.354848 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:45.429544 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:45.429505 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:27:47.538867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:47.538823 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:47.725135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:47.725101 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:47.873612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:47.873398 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:49.382021 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:49.381988 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.007934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:53.007906 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.242014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:53.241981 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.634355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:53.634266 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.791342 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:53.791308 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:53.948294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:53.948248 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:56.523793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.523762 4471 generic.go:356] "Generic (PLEG): container finished" podID="959c2dea-6fbd-40af-8f9e-063113a88298" containerID="ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69" exitCode=0 Oct 31 00:27:56.524329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.523835 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerDied","Data":"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69"} Oct 31 00:27:56.524626 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.524605 4471 scope.go:122] "RemoveContainer" containerID="ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69" Oct 31 00:27:56.526817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.526775 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:56.526939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.526922 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:56.527029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:56.527013 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:27:57.069866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.069833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:57.193700 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.193657 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:27:57.530647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.530609 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerStarted","Data":"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7"} Oct 31 00:27:57.575622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.573908 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:27:57.575949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.575919 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:27:57.576139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:57.576114 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 00:27:58.576553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:58.576105 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:27:58.576553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:58.576200 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 00:27:58.927284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:58.927250 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:59.575555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:59.575006 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:27:59.575555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:27:59.575072 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 00:28:00.573026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:00.572985 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:28:00.573652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:00.573619 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 00:28:01.576976 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:01.576933 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 31 00:28:01.577662 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:01.577625 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 31 00:28:02.576119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:02.576089 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:28:02.576375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:02.576354 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:28:02.576993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:02.576975 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:28:03.349772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:03.349731 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:03.948424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:03.948384 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:04.144335 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:04.144296 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:04.282049 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:28:04.281959 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:28:04.328770 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:04.328741 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:04.504237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:04.504206 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.078031 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:09.077995 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.688463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:09.688427 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.877017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:09.876980 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:10.031248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:10.031163 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:15.193146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:15.193113 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:18.372130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:18.372088 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:18.593151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:18.593110 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:18.994571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:18.994542 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:19.215471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:19.215444 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.271973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:20.271905 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.635721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:20.635641 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:24.882392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:24.882358 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.033837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:25.033799 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.504883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:25.504851 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.656713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:25.656683 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.887858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:25.887767 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.124837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:26.124804 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.194623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:30.194590 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.835065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:30.835020 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:30.991228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:30.991192 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:31.295820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:31.295788 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:33.863056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.863020 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:28:33.866163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.866139 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:33.957006 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.956969 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:33.969720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.969684 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"simulator-w1-982ae3-kserve-self-signed-certs\"" Oct 31 00:28:33.969720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.969688 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"openshift-service-ca.crt\"" Oct 31 00:28:33.969940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.969733 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"kube-root-ca.crt\"" Oct 31 00:28:33.969940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.969770 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-982ae3\"/\"default-dockercfg-jh7nw\"" Oct 31 00:28:33.972504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:33.972484 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:28:34.017750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.017717 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.017950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.017772 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.017950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.017790 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.018039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.017961 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.018039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.017997 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.018147 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.018119 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxc27\" (UniqueName: \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119372 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119439 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119463 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119497 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119538 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.119960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.119590 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxc27\" (UniqueName: \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.120439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.120414 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.123854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.120641 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.123854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.120436 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.131451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.131423 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.131604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.131443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.181963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.181924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxc27\" (UniqueName: \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\") pod \"simulator-w1-982ae3-kserve-54744df6d8-tspnf\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.280898 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:28:34.280861 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:28:34.476967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.476921 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:34.846351 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:28:34.846317 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93cd92b0_686d_47ba_9ab2_d3e72b6e0dd4.slice/crio-edf1f79185dfbbfe6951344efbcd0615b049d195fdaa6928c52914d18e1743ab WatchSource:0}: Error finding container edf1f79185dfbbfe6951344efbcd0615b049d195fdaa6928c52914d18e1743ab: Status 404 returned error can't find the container with id edf1f79185dfbbfe6951344efbcd0615b049d195fdaa6928c52914d18e1743ab Oct 31 00:28:34.847371 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:34.847349 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:28:35.705792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:35.705748 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" event={"ID":"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4","Type":"ContainerStarted","Data":"edf1f79185dfbbfe6951344efbcd0615b049d195fdaa6928c52914d18e1743ab"} Oct 31 00:28:36.293480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:36.293445 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:36.712300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:36.712261 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" event={"ID":"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4","Type":"ContainerStarted","Data":"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f"} Oct 31 00:28:36.830002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:36.829940 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" podStartSLOduration=2.286050829 podStartE2EDuration="3.829927526s" totalImagesPullingTime="1.543876697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:36.829051299 +0000 UTC m=+843.754320251" watchObservedRunningTime="2026-10-31 00:28:36.829927526 +0000 UTC m=+843.755196477" Oct 31 00:28:37.477684 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:37.477648 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:39.059960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:39.059922 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.285567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:41.285500 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.570610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:41.570502 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.902741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:41.902704 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.050317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:42.050282 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.207604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:42.207483 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.477313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:44.477226 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:44.489892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:44.489864 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:44.762427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:44.762348 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:28:46.583632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:46.583597 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:50.729787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.729747 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:28:50.733496 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.733473 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.787931 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.787896 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.787931 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.787933 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.788209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.788055 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.788209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.788091 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmh6q\" (UniqueName: \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.788209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.788119 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.788209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.788147 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.801467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.801425 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:28:50.819074 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.819044 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:50.838139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.838105 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"default-dockercfg-f86h9\"" Oct 31 00:28:50.838295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.838152 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"kube-root-ca.crt\"" Oct 31 00:28:50.838295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.838202 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"openshift-service-ca.crt\"" Oct 31 00:28:50.838295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.838223 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"simulator-w2-e8ba06-kserve-self-signed-certs\"" Oct 31 00:28:50.889363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889326 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889370 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889441 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889470 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmh6q\" (UniqueName: \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889506 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889579 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.889954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889931 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.890034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889950 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.890034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.889972 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.891714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.891690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.891974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.891956 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:50.950694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:50.950657 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmh6q\" (UniqueName: \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\") pod \"simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:51.045578 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.045461 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:51.416001 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:28:51.415971 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37541b57_3039_438c_bebf_f0f56a7db701.slice/crio-5a260edee248647c6dd9d7d97f303a505a1c6d4a41c624352e7234f81f1d4446 WatchSource:0}: Error finding container 5a260edee248647c6dd9d7d97f303a505a1c6d4a41c624352e7234f81f1d4446: Status 404 returned error can't find the container with id 5a260edee248647c6dd9d7d97f303a505a1c6d4a41c624352e7234f81f1d4446 Oct 31 00:28:51.417145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.417118 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:28:51.587702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.587637 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:51.786004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.785903 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" event={"ID":"37541b57-3039-438c-bebf-f0f56a7db701","Type":"ContainerStarted","Data":"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9"} Oct 31 00:28:51.786004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.785953 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" event={"ID":"37541b57-3039-438c-bebf-f0f56a7db701","Type":"ContainerStarted","Data":"5a260edee248647c6dd9d7d97f303a505a1c6d4a41c624352e7234f81f1d4446"} Oct 31 00:28:51.903247 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:51.903198 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" podStartSLOduration=1.903186173 podStartE2EDuration="1.903186173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:51.901825354 +0000 UTC m=+858.827094308" watchObservedRunningTime="2026-10-31 00:28:51.903186173 +0000 UTC m=+858.828455126" Oct 31 00:28:52.046724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:52.046627 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:28:56.602338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:56.602305 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:56.756032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:28:56.755996 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:01.046505 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:01.046470 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:29:01.058890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:01.058864 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:29:01.593341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:01.593311 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:01.844076 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:01.843979 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:29:04.280631 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:29:04.280595 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:29:06.699384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:06.699348 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.177977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.177940 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:29:09.181839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.181822 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.234376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.234346 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"kube-root-ca.crt\"" Oct 31 00:29:09.248905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.248869 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:29:09.265789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.265756 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:09.267997 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.267977 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.268055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.268007 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.268093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.268077 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.268163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.268151 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.268197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.268173 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqg9\" (UniqueName: \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.268303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.268286 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.284329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.284302 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"simulator-w0-101b9e-kserve-self-signed-certs\"" Oct 31 00:29:09.287237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.287218 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"default-dockercfg-c98dn\"" Oct 31 00:29:09.334418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.334386 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-101b9e\"/\"openshift-service-ca.crt\"" Oct 31 00:29:09.368967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.368932 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.368967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.368971 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqg9\" (UniqueName: \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369023 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369051 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369067 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369121 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369496 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369467 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369471 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.369598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.369515 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.371370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.371351 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.371878 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.371856 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.431786 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.431711 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqg9\" (UniqueName: \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\") pod \"simulator-w0-101b9e-kserve-847db75867-tppvc\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.494295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.494262 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:09.867933 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:29:09.867894 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6efb79f1_cdee_4336_936b_2157801f5e3a.slice/crio-9d4604643a96efa1aa6b7ebfa276916d87e992fad61b2bcb21e94feaea1be757 WatchSource:0}: Error finding container 9d4604643a96efa1aa6b7ebfa276916d87e992fad61b2bcb21e94feaea1be757: Status 404 returned error can't find the container with id 9d4604643a96efa1aa6b7ebfa276916d87e992fad61b2bcb21e94feaea1be757 Oct 31 00:29:09.869051 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:09.869019 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:29:10.881038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:10.880998 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" event={"ID":"6efb79f1-cdee-4336-936b-2157801f5e3a","Type":"ContainerStarted","Data":"fcd70532a19cd08cce74edb22fef9ba5b99916c77d8e93988ab3ce9a60597393"} Oct 31 00:29:10.881560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:10.881044 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" event={"ID":"6efb79f1-cdee-4336-936b-2157801f5e3a","Type":"ContainerStarted","Data":"9d4604643a96efa1aa6b7ebfa276916d87e992fad61b2bcb21e94feaea1be757"} Oct 31 00:29:10.998355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:10.998292 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" podStartSLOduration=1.998277697 podStartE2EDuration="1.998277697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:10.996687787 +0000 UTC m=+877.921956741" watchObservedRunningTime="2026-10-31 00:29:10.998277697 +0000 UTC m=+877.923546690" Oct 31 00:29:11.495397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:11.495360 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:11.688466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:11.688435 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:16.687007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:16.686969 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:19.495221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:19.495178 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:19.513809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:19.513780 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:19.940373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:19.940342 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:29:20.086920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:20.086888 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.395618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:21.395581 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:21.587860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:21.587830 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:26.523013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:26.522972 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:31.599772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:31.599717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:33.312315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:33.312280 4471 scope.go:122] "RemoveContainer" containerID="0c08d2eec32b4b117c717905ab5e4d6942af97914ef856f231ac0c39d01a9b85" Oct 31 00:29:34.279963 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:29:34.279924 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:29:34.560684 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.560607 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:29:34.565265 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.565248 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.611962 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.611927 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.611962 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.611961 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.611962 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.611979 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.612290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.612000 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.612290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.612139 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.612290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.612198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdh2\" (UniqueName: \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.660329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.660292 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:34.668472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.668438 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"openshift-service-ca.crt\"" Oct 31 00:29:34.668472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.668466 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"kube-root-ca.crt\"" Oct 31 00:29:34.668679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.668444 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"default-dockercfg-n9zmp\"" Oct 31 00:29:34.668679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.668508 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6f4a5\"/\"premium-w6-a6f4a5-kserve-self-signed-certs\"" Oct 31 00:29:34.670415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.670397 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:29:34.713378 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713341 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdh2\" (UniqueName: \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.713601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713445 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.713601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713476 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.713601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713507 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.713787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713763 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.713861 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.713843 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.714338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.714297 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.714850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.714577 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.714850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.714818 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.717157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.717129 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.717751 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.717731 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.774849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.774818 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdh2\" (UniqueName: \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\") pod \"premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:34.876866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:34.876776 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:35.277516 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:29:35.277488 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b30eb3_3d46_43fa_8fcc_28bfb444ebe3.slice/crio-a62decde01a0f4dbcc32481545c3c1793b9a1b2342ec56b938faa42f50906dda WatchSource:0}: Error finding container a62decde01a0f4dbcc32481545c3c1793b9a1b2342ec56b938faa42f50906dda: Status 404 returned error can't find the container with id a62decde01a0f4dbcc32481545c3c1793b9a1b2342ec56b938faa42f50906dda Oct 31 00:29:35.278895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:35.278870 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:29:36.010128 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:36.010095 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" event={"ID":"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3","Type":"ContainerStarted","Data":"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315"} Oct 31 00:29:36.010562 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:36.010135 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" event={"ID":"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3","Type":"ContainerStarted","Data":"a62decde01a0f4dbcc32481545c3c1793b9a1b2342ec56b938faa42f50906dda"} Oct 31 00:29:36.131830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:36.131774 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" podStartSLOduration=2.131759435 podStartE2EDuration="2.131759435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:36.130001257 +0000 UTC m=+903.055270209" watchObservedRunningTime="2026-10-31 00:29:36.131759435 +0000 UTC m=+903.057028423" Oct 31 00:29:36.675859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:36.675825 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:36.877935 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:36.877896 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:41.614327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:41.614273 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:44.876972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:44.876935 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:44.889136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:44.889106 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:45.065983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:45.065951 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:29:46.577827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:46.577778 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:49.120454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:49.120415 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:51.580727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:51.580689 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:56.643719 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:29:56.643679 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:00.209890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.209855 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:00.213458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.213441 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.249717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.249684 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:00.253093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.253077 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.307852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.307808 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:00.310248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.310218 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:00.362257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.362226 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:30:00.376826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.376788 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.376977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.376848 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.376977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.376951 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjgh\" (UniqueName: \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.377083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.376980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdbf8\" (UniqueName: \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.478598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.478506 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.478598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.478578 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.478831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.478628 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjgh\" (UniqueName: \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.478831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.478652 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdbf8\" (UniqueName: \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.479209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.479188 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.479251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.479208 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.540473 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.540443 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdbf8\" (UniqueName: \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.540473 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.540463 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjgh\" (UniqueName: \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\") pod \"maas-api-key-cleanup-29890110-zsvws\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.565307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.565278 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:00.826179 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.826103 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:00.938363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:00.938325 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:00.938488 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:00.938363 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2bcd4a_65aa_4c2b_b449_c48d9b54e5f8.slice/crio-3617cde1878a5976e133f3e7d06cad392bbf76ba6e98b012aa424d73c9d6a164 WatchSource:0}: Error finding container 3617cde1878a5976e133f3e7d06cad392bbf76ba6e98b012aa424d73c9d6a164: Status 404 returned error can't find the container with id 3617cde1878a5976e133f3e7d06cad392bbf76ba6e98b012aa424d73c9d6a164 Oct 31 00:30:01.134273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:01.134184 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" event={"ID":"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8","Type":"ContainerStarted","Data":"3617cde1878a5976e133f3e7d06cad392bbf76ba6e98b012aa424d73c9d6a164"} Oct 31 00:30:01.191027 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:01.190995 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b74995_2005_4360_a13a_765cda340918.slice/crio-5518925728f6ab5833d7c8e1c6b0171ad55deca4eb72dd68ab105ab5610ce21e WatchSource:0}: Error finding container 5518925728f6ab5833d7c8e1c6b0171ad55deca4eb72dd68ab105ab5610ce21e: Status 404 returned error can't find the container with id 5518925728f6ab5833d7c8e1c6b0171ad55deca4eb72dd68ab105ab5610ce21e Oct 31 00:30:01.192059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:01.192035 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:01.598094 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:01.598065 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:02.139598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:02.139564 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerStarted","Data":"5518925728f6ab5833d7c8e1c6b0171ad55deca4eb72dd68ab105ab5610ce21e"} Oct 31 00:30:04.289130 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:04.289085 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:30:06.162122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:06.162084 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerStarted","Data":"e5b9828908a13f9269a8c8d6a526277191ea79881be7d2ed2b0ba983e445d66b"} Oct 31 00:30:06.280293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:06.280243 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" podStartSLOduration=1.65818723 podStartE2EDuration="6.280228127s" totalImagesPullingTime="4.622040897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:06.278457439 +0000 UTC m=+933.203726391" watchObservedRunningTime="2026-10-31 00:30:06.280228127 +0000 UTC m=+933.205497079" Oct 31 00:30:06.585777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:06.585736 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:07.168599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.168566 4471 generic.go:356] "Generic (PLEG): container finished" podID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerID="2aaa4f60189c815bb5bb1e9b23d54fd2c2d032b637cfe2b8e1c3ebf6c3dbbff0" exitCode=60 Oct 31 00:30:07.169078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.168655 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" event={"ID":"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8","Type":"ContainerDied","Data":"2aaa4f60189c815bb5bb1e9b23d54fd2c2d032b637cfe2b8e1c3ebf6c3dbbff0"} Oct 31 00:30:07.169078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.168871 4471 scope.go:122] "RemoveContainer" containerID="2aaa4f60189c815bb5bb1e9b23d54fd2c2d032b637cfe2b8e1c3ebf6c3dbbff0" Oct 31 00:30:07.170287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.170261 4471 generic.go:356] "Generic (PLEG): container finished" podID="45b74995-2005-4360-a13a-765cda340918" containerID="e5b9828908a13f9269a8c8d6a526277191ea79881be7d2ed2b0ba983e445d66b" exitCode=60 Oct 31 00:30:07.170402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.170346 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerDied","Data":"e5b9828908a13f9269a8c8d6a526277191ea79881be7d2ed2b0ba983e445d66b"} Oct 31 00:30:07.170546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:07.170513 4471 scope.go:122] "RemoveContainer" containerID="e5b9828908a13f9269a8c8d6a526277191ea79881be7d2ed2b0ba983e445d66b" Oct 31 00:30:08.177235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.177199 4471 generic.go:356] "Generic (PLEG): container finished" podID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerID="c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be" exitCode=60 Oct 31 00:30:08.177732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.177271 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" event={"ID":"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8","Type":"ContainerDied","Data":"c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be"} Oct 31 00:30:08.177732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.177349 4471 scope.go:122] "RemoveContainer" containerID="2aaa4f60189c815bb5bb1e9b23d54fd2c2d032b637cfe2b8e1c3ebf6c3dbbff0" Oct 31 00:30:08.177732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.177643 4471 scope.go:122] "RemoveContainer" containerID="c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be" Oct 31 00:30:08.177935 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:08.177914 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n_odh-ai-gateway-infra(bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" Oct 31 00:30:08.179220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.179199 4471 generic.go:356] "Generic (PLEG): container finished" podID="45b74995-2005-4360-a13a-765cda340918" containerID="3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f" exitCode=60 Oct 31 00:30:08.179300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.179272 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerDied","Data":"3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f"} Oct 31 00:30:08.179614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.179597 4471 scope.go:122] "RemoveContainer" containerID="3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f" Oct 31 00:30:08.179820 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:08.179801 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-zsvws_odh-ai-gateway-infra(45b74995-2005-4360-a13a-765cda340918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" podUID="45b74995-2005-4360-a13a-765cda340918" Oct 31 00:30:08.191144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:08.191119 4471 scope.go:122] "RemoveContainer" containerID="e5b9828908a13f9269a8c8d6a526277191ea79881be7d2ed2b0ba983e445d66b" Oct 31 00:30:09.186892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:09.186863 4471 scope.go:122] "RemoveContainer" containerID="c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be" Oct 31 00:30:09.187316 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:09.187171 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n_odh-ai-gateway-infra(bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" Oct 31 00:30:09.188386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:09.188363 4471 scope.go:122] "RemoveContainer" containerID="3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f" Oct 31 00:30:09.188602 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:09.188584 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-zsvws_odh-ai-gateway-infra(45b74995-2005-4360-a13a-765cda340918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" podUID="45b74995-2005-4360-a13a-765cda340918" Oct 31 00:30:11.674798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:11.674765 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:12.952100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:12.952064 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:30:12.956804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:12.956784 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.001583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001513 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.001765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001695 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.001833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001805 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btt2\" (UniqueName: \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.001887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001858 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.001936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001899 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.002003 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.001988 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.043008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.042970 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:13.060868 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.060834 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"premium-w2-e8ba06-kserve-self-signed-certs\"" Oct 31 00:30:13.062770 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.062747 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:30:13.102702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102670 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.102702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102715 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.102974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102764 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.102974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102829 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btt2\" (UniqueName: \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.102974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102872 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.102974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.102904 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.103229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.103201 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.103354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.103235 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.103407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.103385 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.105170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.105141 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.105495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.105467 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.164354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.164320 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btt2\" (UniqueName: \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\") pod \"premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.268419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.268326 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:13.651878 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:13.651848 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e03405_6e4a_444d_9b6f_c661b814b948.slice/crio-a40650888aed3830d49d57ca7280fe1337e5107dcd78b8deed95cc4e21d516ff WatchSource:0}: Error finding container a40650888aed3830d49d57ca7280fe1337e5107dcd78b8deed95cc4e21d516ff: Status 404 returned error can't find the container with id a40650888aed3830d49d57ca7280fe1337e5107dcd78b8deed95cc4e21d516ff Oct 31 00:30:13.653313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:13.653286 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:30:14.220443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:14.220406 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" event={"ID":"06e03405-6e4a-444d-9b6f-c661b814b948","Type":"ContainerStarted","Data":"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad"} Oct 31 00:30:14.220885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:14.220452 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" event={"ID":"06e03405-6e4a-444d-9b6f-c661b814b948","Type":"ContainerStarted","Data":"a40650888aed3830d49d57ca7280fe1337e5107dcd78b8deed95cc4e21d516ff"} Oct 31 00:30:14.268734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:14.268703 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:14.336927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:14.336874 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" podStartSLOduration=2.336859364 podStartE2EDuration="2.336859364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:14.33673471 +0000 UTC m=+941.262003662" watchObservedRunningTime="2026-10-31 00:30:14.336859364 +0000 UTC m=+941.262128317" Oct 31 00:30:15.864219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.864176 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf"] Oct 31 00:30:15.868028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.868004 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:15.921596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.921563 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:30:15.924317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.924289 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf"] Oct 31 00:30:15.929356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.929327 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mlz\" (UniqueName: \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:15.929499 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.929401 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:15.929594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:15.929578 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.031115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.031073 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.031268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.031232 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.031399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.031381 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mlz\" (UniqueName: \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.031883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.031864 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.033644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.033623 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.092550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.092486 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mlz\" (UniqueName: \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\") pod \"maas-api-revoke-sub-1743fd72-6phqf\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.180992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.180953 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:16.550115 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:16.550079 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512be1b4_32f9_4805_8734_db3fe8ae224b.slice/crio-4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe WatchSource:0}: Error finding container 4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe: Status 404 returned error can't find the container with id 4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe Oct 31 00:30:16.552237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.552212 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf"] Oct 31 00:30:16.674543 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:16.674496 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:17.236095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:17.236063 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" event={"ID":"512be1b4-32f9-4805-8734-db3fe8ae224b","Type":"ContainerStarted","Data":"90411060af95da23837ff198b04e7704bc7c866daba443adec8d6378c04922ac"} Oct 31 00:30:17.236497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:17.236115 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" event={"ID":"512be1b4-32f9-4805-8734-db3fe8ae224b","Type":"ContainerStarted","Data":"4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe"} Oct 31 00:30:17.353135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:17.353076 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" podStartSLOduration=2.353063787 podStartE2EDuration="2.353063787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:17.351156735 +0000 UTC m=+944.276425686" watchObservedRunningTime="2026-10-31 00:30:17.353063787 +0000 UTC m=+944.278332739" Oct 31 00:30:18.242156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:18.242120 4471 generic.go:356] "Generic (PLEG): container finished" podID="512be1b4-32f9-4805-8734-db3fe8ae224b" containerID="90411060af95da23837ff198b04e7704bc7c866daba443adec8d6378c04922ac" exitCode=0 Oct 31 00:30:18.242610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:18.242207 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" event={"ID":"512be1b4-32f9-4805-8734-db3fe8ae224b","Type":"ContainerDied","Data":"90411060af95da23837ff198b04e7704bc7c866daba443adec8d6378c04922ac"} Oct 31 00:30:19.373400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.373376 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:19.464829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.464791 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\") pod \"512be1b4-32f9-4805-8734-db3fe8ae224b\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " Oct 31 00:30:19.465001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.464867 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\" (UniqueName: \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\") pod \"512be1b4-32f9-4805-8734-db3fe8ae224b\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " Oct 31 00:30:19.465001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.464916 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\") pod \"512be1b4-32f9-4805-8734-db3fe8ae224b\" (UID: \"512be1b4-32f9-4805-8734-db3fe8ae224b\") " Oct 31 00:30:19.465181 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.465158 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca" pod "512be1b4-32f9-4805-8734-db3fe8ae224b" (UID: "512be1b4-32f9-4805-8734-db3fe8ae224b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:19.466948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.466914 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca" pod "512be1b4-32f9-4805-8734-db3fe8ae224b" (UID: "512be1b4-32f9-4805-8734-db3fe8ae224b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:19.467106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.467089 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz" pod "512be1b4-32f9-4805-8734-db3fe8ae224b" (UID: "512be1b4-32f9-4805-8734-db3fe8ae224b"). InnerVolumeSpecName "kube-api-access-w9mlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:19.565834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.565735 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.565834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.565772 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mlz\" (UniqueName: \"kubernetes.io/projected/512be1b4-32f9-4805-8734-db3fe8ae224b-kube-api-access-w9mlz\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:19.565834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:19.565786 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/512be1b4-32f9-4805-8734-db3fe8ae224b-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.253084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.253048 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" Oct 31 00:30:20.253277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.253082 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf" event={"ID":"512be1b4-32f9-4805-8734-db3fe8ae224b","Type":"ContainerDied","Data":"4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe"} Oct 31 00:30:20.253277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.253142 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcc133cc5a76cf475cd36da1c9c78690434377b1a55efde35b412450a6779fe" Oct 31 00:30:20.499913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.499828 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:20.648609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.648490 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:20.961905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:20.961814 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:21.201971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:21.201932 4471 scope.go:122] "RemoveContainer" containerID="c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be" Oct 31 00:30:21.278558 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:21.278503 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:21.682960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:21.682928 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:22.201944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:22.201908 4471 scope.go:122] "RemoveContainer" containerID="3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f" Oct 31 00:30:22.265474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:22.265442 4471 generic.go:356] "Generic (PLEG): container finished" podID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerID="326ef47166e05f84a13b95a12a82545144c8f4980619232d971ceaaf477a14cd" exitCode=60 Oct 31 00:30:22.265662 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:22.265615 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" event={"ID":"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8","Type":"ContainerDied","Data":"326ef47166e05f84a13b95a12a82545144c8f4980619232d971ceaaf477a14cd"} Oct 31 00:30:22.265754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:22.265673 4471 scope.go:122] "RemoveContainer" containerID="c501ac0d0347580d017cc8071be451640964942c415c9b196e26a996be10e2be" Oct 31 00:30:22.266136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:22.266112 4471 scope.go:122] "RemoveContainer" containerID="326ef47166e05f84a13b95a12a82545144c8f4980619232d971ceaaf477a14cd" Oct 31 00:30:22.267062 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:22.266421 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n_odh-ai-gateway-infra(bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" Oct 31 00:30:23.268504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.268474 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:23.274683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.274653 4471 generic.go:356] "Generic (PLEG): container finished" podID="45b74995-2005-4360-a13a-765cda340918" containerID="d7f8a23f088d689d34ef03ca71eb658d38c4cbe7cf6948ef8bb81ff0deb144e6" exitCode=60 Oct 31 00:30:23.274843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.274723 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerDied","Data":"d7f8a23f088d689d34ef03ca71eb658d38c4cbe7cf6948ef8bb81ff0deb144e6"} Oct 31 00:30:23.274843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.274775 4471 scope.go:122] "RemoveContainer" containerID="3ea04a12c17211ce2bc5953763e3e19f30e32bced6da0263e71b41465371bc7f" Oct 31 00:30:23.275136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.275117 4471 scope.go:122] "RemoveContainer" containerID="d7f8a23f088d689d34ef03ca71eb658d38c4cbe7cf6948ef8bb81ff0deb144e6" Oct 31 00:30:23.275388 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:23.275369 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890110-zsvws_odh-ai-gateway-infra(45b74995-2005-4360-a13a-765cda340918)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" podUID="45b74995-2005-4360-a13a-765cda340918" Oct 31 00:30:23.282263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.282242 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:23.394337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.394307 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:23.543783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.543757 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:23.613507 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.613472 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\") pod \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " Oct 31 00:30:23.613726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.613704 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\" (UniqueName: \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\") pod \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\" (UID: \"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8\") " Oct 31 00:30:23.613944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.613920 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca" pod "bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" (UID: "bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.614078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.614063 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.615910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.615884 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8" pod "bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" (UID: "bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8"). InnerVolumeSpecName "kube-api-access-sdbf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.715474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:23.715382 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdbf8\" (UniqueName: \"kubernetes.io/projected/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8-kube-api-access-sdbf8\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:24.280514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.280481 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:30:24.282111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.282092 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" Oct 31 00:30:24.282244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.282109 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n" event={"ID":"bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8","Type":"ContainerDied","Data":"3617cde1878a5976e133f3e7d06cad392bbf76ba6e98b012aa424d73c9d6a164"} Oct 31 00:30:24.282244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.282176 4471 scope.go:122] "RemoveContainer" containerID="326ef47166e05f84a13b95a12a82545144c8f4980619232d971ceaaf477a14cd" Oct 31 00:30:24.448195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.448136 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:24.512674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.512641 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:24.521502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.521470 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890110-hkf4n"] Oct 31 00:30:24.592764 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.592736 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:24.726986 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.726948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\") pod \"45b74995-2005-4360-a13a-765cda340918\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " Oct 31 00:30:24.727167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.727022 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\" (UniqueName: \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\") pod \"45b74995-2005-4360-a13a-765cda340918\" (UID: \"45b74995-2005-4360-a13a-765cda340918\") " Oct 31 00:30:24.727375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.727350 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca" pod "45b74995-2005-4360-a13a-765cda340918" (UID: "45b74995-2005-4360-a13a-765cda340918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:24.729090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.729068 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh" pod "45b74995-2005-4360-a13a-765cda340918" (UID: "45b74995-2005-4360-a13a-765cda340918"). InnerVolumeSpecName "kube-api-access-7sjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:24.828375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.828282 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjgh\" (UniqueName: \"kubernetes.io/projected/45b74995-2005-4360-a13a-765cda340918-kube-api-access-7sjgh\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:24.828375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:24.828316 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b74995-2005-4360-a13a-765cda340918-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:25.207426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.207391 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" path="/var/lib/kubelet/pods/bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8/volumes" Oct 31 00:30:25.291659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.291627 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" event={"ID":"45b74995-2005-4360-a13a-765cda340918","Type":"ContainerDied","Data":"5518925728f6ab5833d7c8e1c6b0171ad55deca4eb72dd68ab105ab5610ce21e"} Oct 31 00:30:25.292105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.291672 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws" Oct 31 00:30:25.292105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.291687 4471 scope.go:122] "RemoveContainer" containerID="d7f8a23f088d689d34ef03ca71eb658d38c4cbe7cf6948ef8bb81ff0deb144e6" Oct 31 00:30:25.456365 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.456322 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:25.504712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:25.504616 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890110-zsvws"] Oct 31 00:30:26.675079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:26.675048 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.209061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.209031 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b74995-2005-4360-a13a-765cda340918" path="/var/lib/kubelet/pods/45b74995-2005-4360-a13a-765cda340918/volumes" Oct 31 00:30:27.688784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.688757 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 00:30:27.689161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689147 4471 cpu_manager.go:414] "containerMap: removing container" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689162 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689177 4471 cpu_manager.go:414] "containerMap: removing container" podUID="512be1b4-32f9-4805-8734-db3fe8ae224b" containerName="revoke-keys" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689182 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512be1b4-32f9-4805-8734-db3fe8ae224b" containerName="revoke-keys" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689195 4471 cpu_manager.go:414] "containerMap: removing container" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689199 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689207 4471 cpu_manager.go:414] "containerMap: removing container" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689212 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689227 4471 cpu_manager.go:414] "containerMap: removing container" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689232 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689238 4471 cpu_manager.go:414] "containerMap: removing container" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689242 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689302 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689309 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="512be1b4-32f9-4805-8734-db3fe8ae224b" containerName="revoke-keys" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689317 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689324 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689333 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2bcd4a-65aa-4c2b-b449-c48d9b54e5f8" containerName="cleanup" Oct 31 00:30:27.689493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.689338 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:27.692863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.692845 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.785074 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.785036 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:27.797240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.797205 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"openshift-service-ca.crt\"" Oct 31 00:30:27.797402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.797264 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"kube-root-ca.crt\"" Oct 31 00:30:27.797402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.797320 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"default-dockercfg-zzljb\"" Oct 31 00:30:27.797402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.797334 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"premium-w4-da9d07-kserve-self-signed-certs\"" Oct 31 00:30:27.799288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.799269 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 00:30:27.858677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858646 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.858677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858682 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.858896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858717 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.858896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858793 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwfq\" (UniqueName: \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.858896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858834 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.858990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.858970 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960040 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960080 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960108 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960139 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960157 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwfq\" (UniqueName: \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960214 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960545 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960621 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960588 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.960686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.960615 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.962587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.962562 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:27.963170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:27.963144 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:28.022301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:28.022262 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwfq\" (UniqueName: \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\") pod \"premium-w4-da9d07-kserve-858f54dcbc-bhrdq\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:28.303793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:28.303704 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:28.674031 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:28.673990 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8129dfeb_bcb3_4c72_844d_cee5956a8da6.slice/crio-c0812c6753fdf24c65f81a52e6698a621fd1ee895c987d1edae94685bd29289d WatchSource:0}: Error finding container c0812c6753fdf24c65f81a52e6698a621fd1ee895c987d1edae94685bd29289d: Status 404 returned error can't find the container with id c0812c6753fdf24c65f81a52e6698a621fd1ee895c987d1edae94685bd29289d Oct 31 00:30:28.675859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:28.675836 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 00:30:29.314755 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:29.314716 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" event={"ID":"8129dfeb-bcb3-4c72-844d-cee5956a8da6","Type":"ContainerStarted","Data":"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced"} Oct 31 00:30:29.314755 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:29.314762 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" event={"ID":"8129dfeb-bcb3-4c72-844d-cee5956a8da6","Type":"ContainerStarted","Data":"c0812c6753fdf24c65f81a52e6698a621fd1ee895c987d1edae94685bd29289d"} Oct 31 00:30:29.431268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:29.431214 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" podStartSLOduration=2.431200419 podStartE2EDuration="2.431200419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:29.430401843 +0000 UTC m=+956.355670797" watchObservedRunningTime="2026-10-31 00:30:29.431200419 +0000 UTC m=+956.356469371" Oct 31 00:30:30.304444 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:30.304409 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:31.666820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:31.666748 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:34.282455 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:30:34.282411 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:30:36.573840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:36.573808 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:38.304437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:38.304396 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:38.320794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:38.320765 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:38.373548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:38.373497 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 00:30:38.695471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:38.695435 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:41.659449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:41.659407 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:46.057947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.057912 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:30:46.058508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.058382 4471 cpu_manager.go:414] "containerMap: removing container" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:46.058508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.058402 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:46.058508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.058491 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b74995-2005-4360-a13a-765cda340918" containerName="cleanup" Oct 31 00:30:46.063668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.063639 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.122710 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.122676 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:30:46.146305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.146276 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:46.147483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147463 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.147555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147503 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.147627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147608 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.147662 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147640 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.147749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147732 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhbp\" (UniqueName: \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.147798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.147786 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.154654 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.154633 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"openshift-service-ca.crt\"" Oct 31 00:30:46.167329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.167307 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"kube-root-ca.crt\"" Oct 31 00:30:46.169184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.169170 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"premium-w5-412743-kserve-self-signed-certs\"" Oct 31 00:30:46.169427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.169408 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-412743\"/\"default-dockercfg-z8bt4\"" Oct 31 00:30:46.248765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248734 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.248960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248794 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.248960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248818 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.248960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248850 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.248960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248876 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.248960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.248951 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhbp\" (UniqueName: \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.249342 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.249317 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.249474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.249362 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.249559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.249472 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.251720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.251698 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.252167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.252145 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.313249 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.313176 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhbp\" (UniqueName: \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\") pod \"premium-w5-412743-kserve-7c89d65946-ftpkc\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.379159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.379122 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:46.617513 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.617419 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:46.770502 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:30:46.770468 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f11ac15_675a_4de5_be97_fe932e5fa986.slice/crio-ca052fabe708146896555f4457c5bf1bb812430dcbb108a1c4e2e9dabb0a1bd6 WatchSource:0}: Error finding container ca052fabe708146896555f4457c5bf1bb812430dcbb108a1c4e2e9dabb0a1bd6: Status 404 returned error can't find the container with id ca052fabe708146896555f4457c5bf1bb812430dcbb108a1c4e2e9dabb0a1bd6 Oct 31 00:30:46.771929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:46.771902 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:30:47.409957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:47.409916 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" event={"ID":"5f11ac15-675a-4de5-be97-fe932e5fa986","Type":"ContainerStarted","Data":"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768"} Oct 31 00:30:47.410400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:47.409966 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" event={"ID":"5f11ac15-675a-4de5-be97-fe932e5fa986","Type":"ContainerStarted","Data":"ca052fabe708146896555f4457c5bf1bb812430dcbb108a1c4e2e9dabb0a1bd6"} Oct 31 00:30:47.527781 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:47.527722 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" podStartSLOduration=1.527707281 podStartE2EDuration="1.527707281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:47.525895584 +0000 UTC m=+974.451164537" watchObservedRunningTime="2026-10-31 00:30:47.527707281 +0000 UTC m=+974.452976269" Oct 31 00:30:48.380114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:48.380080 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:51.607285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:51.607246 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:56.379772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:56.379733 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:56.391908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:56.391879 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:56.467232 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:56.467206 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:30:56.644260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:30:56.644215 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:01.593351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:01.593315 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:04.280377 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:31:04.280337 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:31:06.572127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:06.572092 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:11.580118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:11.580085 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:16.479693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:16.479644 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.293936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:21.293899 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:21.491588 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:21.491554 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:26.378584 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:26.378542 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:29.325112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:29.325070 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:31.344356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:31.344308 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:34.280304 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:31:34.280264 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:31:36.388080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:36.388021 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:41.386379 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:41.386337 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:46.417618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:46.417500 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:51.399867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:51.399816 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:56.380996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:31:56.380963 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:01.290035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:01.290000 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:02.508525 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:02.508493 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:04.280505 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:32:04.280456 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:32:06.388629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:06.388597 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:11.569802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:11.569763 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:16.593436 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:16.593397 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:21.581801 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:21.581764 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:21.727706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:21.727664 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:26.509423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:26.509382 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:31.536832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:31.536800 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:34.280850 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:32:34.280804 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:32:36.482124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:36.482087 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:40.602395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.602360 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:32:40.606173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.606157 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.658836 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.658801 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a6f4a5-openshift-default-dockercfg-r4d7m\"" Oct 31 00:32:40.710349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.710315 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:32:40.711901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.711881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.711954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.711933 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712011 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712034 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712091 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712066 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712149 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712129 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dww\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712182 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712266 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.712318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.712297 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813311 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813412 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813439 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813463 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dww\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813493 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813566 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813589 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813626 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813769 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.813939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813884 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.814011 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.813991 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.814112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.814090 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.814409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.814393 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.815920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.815897 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.816077 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.816059 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.874629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.874554 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.874923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.874900 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dww\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:40.918959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:40.918918 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:41.337596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.337563 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:32:41.368097 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:32:41.368062 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03880da0_5a1c_4d49_8a4f_eb28c35dd9e8.slice/crio-982ee0263a90b4489286a2a00b7fe540909b732043f9627f5446d175f678d1b4 WatchSource:0}: Error finding container 982ee0263a90b4489286a2a00b7fe540909b732043f9627f5446d175f678d1b4: Status 404 returned error can't find the container with id 982ee0263a90b4489286a2a00b7fe540909b732043f9627f5446d175f678d1b4 Oct 31 00:32:41.370437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.370402 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:41.370556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.370471 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:41.370556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.370504 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:41.570827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.570789 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:41.995195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.995158 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" event={"ID":"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8","Type":"ContainerStarted","Data":"7e91cfb86f4f5922225861432be59f0bfd452c43399dbe9f33810c429f2bf3a8"} Oct 31 00:32:41.995648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:41.995220 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" event={"ID":"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8","Type":"ContainerStarted","Data":"982ee0263a90b4489286a2a00b7fe540909b732043f9627f5446d175f678d1b4"} Oct 31 00:32:42.118924 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:42.118854 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podStartSLOduration=2.118836539 podStartE2EDuration="2.118836539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:42.115873285 +0000 UTC m=+1089.041142238" watchObservedRunningTime="2026-10-31 00:32:42.118836539 +0000 UTC m=+1089.044105494" Oct 31 00:32:42.919715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:42.919682 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:42.920999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:42.920978 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:32:42.921055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:42.921035 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:32:43.920163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:43.920128 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:32:43.920617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:43.920197 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:32:44.919711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:44.919669 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:32:44.919894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:44.919732 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:32:45.920063 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:45.920017 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:32:45.920599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:45.920084 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:32:46.733545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:46.733508 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:46.919727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:46.919690 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 00:32:46.919903 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:46.919770 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 00:32:48.020457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:48.020421 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:48.020949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:48.020800 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:48.021594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:48.021571 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:32:50.086856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:50.086824 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:50.239463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:50.239428 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.783568 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:51.783521 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:52.870347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:52.870308 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:53.042361 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:53.042331 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:56.379718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.379684 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 00:32:56.383205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.383188 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411057 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411102 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411228 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411251 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411289 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7bc\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411376 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411419 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411496 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.411618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.411598 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.435989 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.435958 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-da9d07-openshift-default-dockercfg-ckgpr\"" Oct 31 00:32:56.487766 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.487730 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 00:32:56.512753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.512719 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.512753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.512751 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.512777 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7bc\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.512892 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.512929 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513000 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513072 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513110 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513148 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513151 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513452 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513668 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.513765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513731 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.514015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.513997 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.515216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.515194 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.515467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.515449 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.574519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.574484 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.574679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.574498 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7bc\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.696938 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.696900 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:56.806213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:56.806144 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:57.116164 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:32:57.116133 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e1a669_b28b_4878_8e99_59ee07982fce.slice/crio-a5e5c266d11ce0a84e4741f68f0d7d730eff02c3f989f80d8e3ac7b883e8b925 WatchSource:0}: Error finding container a5e5c266d11ce0a84e4741f68f0d7d730eff02c3f989f80d8e3ac7b883e8b925: Status 404 returned error can't find the container with id a5e5c266d11ce0a84e4741f68f0d7d730eff02c3f989f80d8e3ac7b883e8b925 Oct 31 00:32:57.117705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:57.117682 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 00:32:57.118664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:57.118637 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:57.118762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:57.118700 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:57.118762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:57.118731 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:32:58.098188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.098140 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" event={"ID":"e2e1a669-b28b-4878-8e99-59ee07982fce","Type":"ContainerStarted","Data":"ceb2cf16ad7cd5103967c343c57e0481443500cef85cc3d97bbb4ae32e2354b1"} Oct 31 00:32:58.098759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.098243 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" event={"ID":"e2e1a669-b28b-4878-8e99-59ee07982fce","Type":"ContainerStarted","Data":"a5e5c266d11ce0a84e4741f68f0d7d730eff02c3f989f80d8e3ac7b883e8b925"} Oct 31 00:32:58.278552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.278386 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podStartSLOduration=2.278369467 podStartE2EDuration="2.278369467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:58.278033915 +0000 UTC m=+1105.203302870" watchObservedRunningTime="2026-10-31 00:32:58.278369467 +0000 UTC m=+1105.203638420" Oct 31 00:32:58.697625 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.697584 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:32:58.699244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.699219 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:32:58.699367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:58.699277 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 00:32:59.698183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:59.698139 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:32:59.698630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:32:59.698209 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 00:33:00.698162 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:00.698125 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:33:00.698342 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:00.698197 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 00:33:01.697447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:01.697414 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:33:01.697656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:01.697467 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 00:33:01.926142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:01.926103 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:02.697367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:02.697330 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 00:33:02.697564 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:02.697397 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 00:33:03.701892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:03.701860 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:33:03.702302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:03.702149 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:33:03.702871 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:03.702850 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 00:33:04.280864 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:33:04.280822 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:33:06.994402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:06.994369 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:11.546794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:11.546756 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:12.002785 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:12.002755 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:17.024435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:17.024351 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:21.987372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:21.987337 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.135822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:22.135785 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.321125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:22.321020 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:24.580230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.580195 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 00:33:24.584030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.583996 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.643754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.643723 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 00:33:24.668565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.668507 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:24.687712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.687679 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"distinct-w4-da9d07-kserve-self-signed-certs\"" Oct 31 00:33:24.739932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.739892 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.740130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.739943 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.740130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.740004 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d692p\" (UniqueName: \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.740130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.740037 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.740281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.740153 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.740281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.740242 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841357 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841407 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841429 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d692p\" (UniqueName: \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841456 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841507 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841691 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841605 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.841970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.841955 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.842139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.842127 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.842652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.842624 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.844637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.844613 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.846766 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.846739 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:24.907948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:24.907910 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d692p\" (UniqueName: \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\") pod \"distinct-w4-da9d07-kserve-74c65d4d58-6fvdm\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:25.196000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:25.195959 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:25.567506 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:33:25.567476 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9153ac1_eba6_4adc_8cdd_2c42910be5ad.slice/crio-5846cd8cb1587cf60b708a0f832a69dc70a64ea08530a9f8291ec445024293d1 WatchSource:0}: Error finding container 5846cd8cb1587cf60b708a0f832a69dc70a64ea08530a9f8291ec445024293d1: Status 404 returned error can't find the container with id 5846cd8cb1587cf60b708a0f832a69dc70a64ea08530a9f8291ec445024293d1 Oct 31 00:33:25.568032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:25.568009 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 00:33:26.253000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:26.252962 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" event={"ID":"d9153ac1-eba6-4adc-8cdd-2c42910be5ad","Type":"ContainerStarted","Data":"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763"} Oct 31 00:33:26.253403 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:26.253010 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" event={"ID":"d9153ac1-eba6-4adc-8cdd-2c42910be5ad","Type":"ContainerStarted","Data":"5846cd8cb1587cf60b708a0f832a69dc70a64ea08530a9f8291ec445024293d1"} Oct 31 00:33:26.371484 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:26.371422 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" podStartSLOduration=2.371408624 podStartE2EDuration="2.371408624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:26.369551524 +0000 UTC m=+1133.294820483" watchObservedRunningTime="2026-10-31 00:33:26.371408624 +0000 UTC m=+1133.296677576" Oct 31 00:33:26.966595 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:26.966559 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:27.196260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:27.196220 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:31.968510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:31.968472 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:34.283241 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:33:34.283201 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:33:34.974032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:34.973989 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.196977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:35.196929 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:35.211502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:35.211461 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:35.313458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:35.313376 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 00:33:36.993475 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:36.993441 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:41.963469 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:41.963428 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:47.129889 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:47.129853 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.811610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.811574 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:33:48.816505 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.816482 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.876972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.876934 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:33:48.896245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.896212 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:48.904568 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.904502 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.904759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.904730 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pjk\" (UniqueName: \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.904886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.904774 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.904886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.904859 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.904982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.904960 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.905077 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.905058 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:48.923794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:48.923734 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e8ba06\"/\"distinct-2-w2-e8ba06-kserve-self-signed-certs\"" Oct 31 00:33:49.006002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.005967 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006030 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pjk\" (UniqueName: \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006054 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006098 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006115 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006172 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006505 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006515 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.006624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.006576 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.008278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.008253 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.008603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.008584 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.071226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.071148 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pjk\" (UniqueName: \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\") pod \"distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.128216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.128177 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:49.529372 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:33:49.529342 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2309d5f3_0749_4c51_98d3_5781cbb0f3f6.slice/crio-7d470efe346bb40485704f762e7b4e74a935945e9e6f9e628267857ff33bd53c WatchSource:0}: Error finding container 7d470efe346bb40485704f762e7b4e74a935945e9e6f9e628267857ff33bd53c: Status 404 returned error can't find the container with id 7d470efe346bb40485704f762e7b4e74a935945e9e6f9e628267857ff33bd53c Oct 31 00:33:49.530917 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:49.530894 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:33:50.386465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:50.386422 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" event={"ID":"2309d5f3-0749-4c51-98d3-5781cbb0f3f6","Type":"ContainerStarted","Data":"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77"} Oct 31 00:33:50.386881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:50.386477 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" event={"ID":"2309d5f3-0749-4c51-98d3-5781cbb0f3f6","Type":"ContainerStarted","Data":"7d470efe346bb40485704f762e7b4e74a935945e9e6f9e628267857ff33bd53c"} Oct 31 00:33:50.505488 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:50.505430 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" podStartSLOduration=2.505415062 podStartE2EDuration="2.505415062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:50.503426011 +0000 UTC m=+1157.428694964" watchObservedRunningTime="2026-10-31 00:33:50.505415062 +0000 UTC m=+1157.430684014" Oct 31 00:33:51.129323 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:51.129288 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:52.190997 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:52.190937 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:57.188587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:57.188518 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:58.916405 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:58.916365 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:59.128975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:59.128937 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:59.142496 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:59.142464 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:33:59.444393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:33:59.444365 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:34:02.191759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:02.191722 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:02.442512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:02.442405 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:04.282445 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:34:04.282403 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:34:07.178291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:07.178248 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:12.214014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:12.213978 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:17.188674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:17.188638 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:22.211013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:22.210982 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:23.184294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:23.184256 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:27.186269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:27.186195 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:27.702164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:27.702123 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:32.284207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:32.284148 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:34.281332 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:34:34.281285 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:34:37.268798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:37.268765 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:42.190546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:42.190466 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:47.304402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:47.304307 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:52.401581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:52.401538 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:57.630714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:57.630670 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:58.372164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:34:58.372131 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:01.193262 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:01.193218 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:02.599979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:02.599937 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:04.281127 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:35:04.281087 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:35:07.494954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:07.494915 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:12.487043 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:12.487001 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:15.482955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.482918 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:35:15.488540 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.488504 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.544570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.544503 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:35:15.572931 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.572891 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:15.583987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.583961 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.584071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.583993 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.584120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.584079 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.584153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.584133 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qtc\" (UniqueName: \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.584238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.584219 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.584278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.584263 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.590479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.590459 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"kube-root-ca.crt\"" Oct 31 00:35:15.590580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.590504 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"openshift-service-ca.crt\"" Oct 31 00:35:15.590732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.590719 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"default-dockercfg-jzlwr\"" Oct 31 00:35:15.590806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.590792 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-1e5e75\"/\"distinct-2-w3-1e5e75-kserve-self-signed-certs\"" Oct 31 00:35:15.685203 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685203 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685208 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685398 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qtc\" (UniqueName: \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685485 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685518 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685656 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685688 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.685928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.685906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.687705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.687677 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.687990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.687972 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.748540 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.748451 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qtc\" (UniqueName: \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\") pod \"distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:15.800455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:15.800416 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:16.190028 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:35:16.189991 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded5849bf_a0ae_46a0_bc80_adbe6bedc03c.slice/crio-cd085c1ace40bd1da67884e385e174e6cd86d400271d250077729e7186e9cbeb WatchSource:0}: Error finding container cd085c1ace40bd1da67884e385e174e6cd86d400271d250077729e7186e9cbeb: Status 404 returned error can't find the container with id cd085c1ace40bd1da67884e385e174e6cd86d400271d250077729e7186e9cbeb Oct 31 00:35:16.191355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:16.191328 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:35:16.906266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:16.906228 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" event={"ID":"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c","Type":"ContainerStarted","Data":"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04"} Oct 31 00:35:16.906266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:16.906270 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" event={"ID":"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c","Type":"ContainerStarted","Data":"cd085c1ace40bd1da67884e385e174e6cd86d400271d250077729e7186e9cbeb"} Oct 31 00:35:17.024418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:17.024356 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" podStartSLOduration=2.024340401 podStartE2EDuration="2.024340401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:17.022622035 +0000 UTC m=+1243.947890984" watchObservedRunningTime="2026-10-31 00:35:17.024340401 +0000 UTC m=+1243.949609745" Oct 31 00:35:17.464102 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:17.464060 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:17.801027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:17.800927 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:20.060283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:20.060235 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf"] Oct 31 00:35:20.109624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:20.109591 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1743fd72-6phqf"] Oct 31 00:35:21.208415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:21.208379 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512be1b4-32f9-4805-8734-db3fe8ae224b" path="/var/lib/kubelet/pods/512be1b4-32f9-4805-8734-db3fe8ae224b/volumes" Oct 31 00:35:22.591270 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:22.591235 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:22.735897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:22.735848 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:25.801591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:25.801547 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:25.814454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:25.814425 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:25.963641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:25.963613 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:35:27.594126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:27.594091 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:28.340376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.340338 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 00:35:28.344136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.344119 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.429098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.429036 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:28.435692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.435657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.435845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.435710 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.435914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.435886 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.436015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.435931 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.436200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.436023 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95bm\" (UniqueName: \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.436310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.436292 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.448341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.448316 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-da9d07\"/\"distinct-2-w4-da9d07-kserve-self-signed-certs\"" Oct 31 00:35:28.450567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.450547 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 00:35:28.537943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.537905 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.537943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.537953 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95bm\" (UniqueName: \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.537984 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538161 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538221 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538295 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538502 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538513 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.538658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.538641 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.540326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.540305 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.540610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.540591 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.601858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.601782 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95bm\" (UniqueName: \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\") pod \"distinct-2-w4-da9d07-kserve-54457fd465-r7rgw\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:28.658225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:28.658175 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:29.064979 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:35:29.064950 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod336d9b30_364f_4901_9337_ac3fdd11b18e.slice/crio-3b3512fb92acf185973effe7daf5a6c0ba3e46abdf31d9535d03d0aa5a440828 WatchSource:0}: Error finding container 3b3512fb92acf185973effe7daf5a6c0ba3e46abdf31d9535d03d0aa5a440828: Status 404 returned error can't find the container with id 3b3512fb92acf185973effe7daf5a6c0ba3e46abdf31d9535d03d0aa5a440828 Oct 31 00:35:29.067029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:29.067006 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 00:35:29.979017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:29.978982 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" event={"ID":"336d9b30-364f-4901-9337-ac3fdd11b18e","Type":"ContainerStarted","Data":"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911"} Oct 31 00:35:29.979412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:29.979030 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" event={"ID":"336d9b30-364f-4901-9337-ac3fdd11b18e","Type":"ContainerStarted","Data":"3b3512fb92acf185973effe7daf5a6c0ba3e46abdf31d9535d03d0aa5a440828"} Oct 31 00:35:30.097590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:30.097513 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" podStartSLOduration=2.097494329 podStartE2EDuration="2.097494329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:30.095184611 +0000 UTC m=+1257.020453580" watchObservedRunningTime="2026-10-31 00:35:30.097494329 +0000 UTC m=+1257.022763285" Oct 31 00:35:30.659176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:30.659138 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:32.580833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:32.580794 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:33.514423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:33.514385 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:34.281505 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:35:34.281470 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:35:37.599191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:37.599152 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:38.659282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:38.659249 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:38.671821 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:38.671796 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:39.041126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:39.041048 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 00:35:42.598841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:42.598803 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:47.613848 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:47.613807 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:49.229449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:49.229419 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:52.869382 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:52.869346 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:56.775017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.774980 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:35:56.778949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.778921 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.831196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.831167 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e8ba06-openshift-default-dockercfg-97jbg\"" Oct 31 00:35:56.878909 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.878878 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.878916 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879023 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879061 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879089 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879168 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879318 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879398 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.879447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.879429 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4r9d\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.883111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.883087 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:35:56.980756 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980718 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.980956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980766 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4r9d\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.980956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980821 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.980956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.980956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980911 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.980956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980939 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.980977 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981006 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981095 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981219 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981221 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981324 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981441 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.981828 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.981808 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.983300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.983276 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:56.983399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:56.983383 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:57.042177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.042102 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:57.042177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.042121 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4r9d\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:57.092578 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.092543 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:57.516555 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:35:57.516510 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e7f5da_c5cd_48be_82ea_ff89e53d6e3e.slice/crio-4e60a3d2ff6e8ffcd09b36b89fc07105b111c75dae9caab27df1de6d4f325888 WatchSource:0}: Error finding container 4e60a3d2ff6e8ffcd09b36b89fc07105b111c75dae9caab27df1de6d4f325888: Status 404 returned error can't find the container with id 4e60a3d2ff6e8ffcd09b36b89fc07105b111c75dae9caab27df1de6d4f325888 Oct 31 00:35:57.517849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.517817 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:35:57.519050 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.519014 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:35:57.519130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.519083 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:35:57.519130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.519112 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:35:57.881726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:57.881652 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:58.136687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:58.136596 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" event={"ID":"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e","Type":"ContainerStarted","Data":"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8"} Oct 31 00:35:58.136687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:58.136659 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" event={"ID":"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e","Type":"ContainerStarted","Data":"4e60a3d2ff6e8ffcd09b36b89fc07105b111c75dae9caab27df1de6d4f325888"} Oct 31 00:35:59.092942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:59.092906 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:35:59.094674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:59.094646 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:35:59.094799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:35:59.094697 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:00.100348 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:00.100272 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:36:00.101011 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:00.100383 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:01.093119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:01.093085 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:36:01.093301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:01.093142 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:01.455692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:01.455661 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:01.602030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:01.601994 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:02.093301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:02.093266 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:36:02.093493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:02.093321 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:02.878060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:02.878019 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:03.093887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:03.093851 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:36:03.094077 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:03.093910 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:04.093216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:04.093166 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 31 00:36:04.093696 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:04.093247 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 31 00:36:04.286924 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:36:04.286881 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:36:05.098223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:05.098186 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:36:05.098628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:05.098562 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:36:05.099193 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:05.099173 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:36:05.275614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:05.275545 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podStartSLOduration=9.275513043 podStartE2EDuration="9.275513043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:58.281808084 +0000 UTC m=+1285.207077038" watchObservedRunningTime="2026-10-31 00:36:05.275513043 +0000 UTC m=+1292.200781998" Oct 31 00:36:08.006357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:08.006319 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:12.286214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.286180 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:36:12.290371 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.290340 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352204 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352243 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352348 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352407 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352456 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352579 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352474 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352594 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352619 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z8s\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.352706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.352688 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.375619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.375566 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:36:12.453341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453306 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453362 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453382 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453408 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453425 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z8s\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453459 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453510 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453846 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453565 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453846 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453654 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453854 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.453948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.453903 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.454108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.454090 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.454192 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.454173 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.454235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.454202 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.455753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.455734 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.456416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.456398 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.514690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.514663 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.514825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.514788 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z8s\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\") pod \"e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:12.603705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:12.603613 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:13.025680 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:36:13.025636 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37f42c5_9658_4fdc_b34f_be9eec95403a.slice/crio-b16eb96829473c32d3a46cac4238d409f599eb88348ff6d6c196061179345152 WatchSource:0}: Error finding container b16eb96829473c32d3a46cac4238d409f599eb88348ff6d6c196061179345152: Status 404 returned error can't find the container with id b16eb96829473c32d3a46cac4238d409f599eb88348ff6d6c196061179345152 Oct 31 00:36:13.027058 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.027030 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:36:13.028096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.028068 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:13.028165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.028131 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:13.028165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.028161 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:13.055285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.055255 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:13.239475 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.239433 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" event={"ID":"a37f42c5-9658-4fdc-b34f-be9eec95403a","Type":"ContainerStarted","Data":"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420"} Oct 31 00:36:13.239658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.239512 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" event={"ID":"a37f42c5-9658-4fdc-b34f-be9eec95403a","Type":"ContainerStarted","Data":"b16eb96829473c32d3a46cac4238d409f599eb88348ff6d6c196061179345152"} Oct 31 00:36:13.363316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.363175 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podStartSLOduration=1.363161789 podStartE2EDuration="1.363161789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:13.362340299 +0000 UTC m=+1300.287609253" watchObservedRunningTime="2026-10-31 00:36:13.363161789 +0000 UTC m=+1300.288430741" Oct 31 00:36:13.604292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:13.604254 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:14.605676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:14.605643 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:14.606154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:14.605697 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:15.604051 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:15.604014 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:15.604252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:15.604074 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:16.604218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:16.604176 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:16.604766 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:16.604242 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:17.604480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:17.604446 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:17.604892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:17.604512 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:17.977221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:17.977131 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:18.604612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:18.604575 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:18.605022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:18.604646 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:19.604500 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:19.604463 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 31 00:36:19.604701 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:19.604520 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 31 00:36:20.717295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:20.717258 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:20.719469 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:20.719443 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:20.720978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:20.720951 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:36:21.686620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:21.686576 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:36:21.687019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:21.686861 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" containerID="cri-o://0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9" gracePeriod=30 Oct 31 00:36:22.972969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:22.972934 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:23.118339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:23.118309 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:23.359891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:23.359805 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:25.237727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:25.237687 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c2j7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:36:25.238221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:25.237766 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:36:26.953497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:26.953430 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:36:27.043163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043129 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043169 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043202 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043333 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043360 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043394 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043415 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043452 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043491 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\") pod \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\" (UID: \"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e\") " Oct 31 00:36:27.043640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043549 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:27.043836 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043686 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:27.043903 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.043884 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:27.044081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.044002 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.044081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.044027 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.044081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.044040 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.044301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.044088 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:27.044356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.044339 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:27.046151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.046110 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:36:27.046268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.046159 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:27.046327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.046281 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:36:27.046503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.046481 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7" pod "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" (UID: "daa66a9f-1f63-4b29-8ce1-7ae32d47a96e"). InnerVolumeSpecName "kube-api-access-dmtv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:27.119885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.119846 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:27.145706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145669 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.145706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145698 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.145706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145709 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmtv7\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-kube-api-access-dmtv7\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.145921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145721 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.145921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145730 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.145921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.145738 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:27.370611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.370570 4471 generic.go:356] "Generic (PLEG): container finished" podID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerID="0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9" exitCode=0 Oct 31 00:36:27.370809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.370660 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" Oct 31 00:36:27.370809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.370698 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" event={"ID":"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e","Type":"ContainerDied","Data":"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9"} Oct 31 00:36:27.370809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.370767 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f" event={"ID":"daa66a9f-1f63-4b29-8ce1-7ae32d47a96e","Type":"ContainerDied","Data":"4a8af970a820cae79c7b2ec3b725a1a81e880d11314ea190ca76c8a868923acb"} Oct 31 00:36:27.370809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.370800 4471 scope.go:122] "RemoveContainer" containerID="0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9" Oct 31 00:36:27.382131 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.382109 4471 scope.go:122] "RemoveContainer" containerID="0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9" Oct 31 00:36:27.382483 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:36:27.382456 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9\": container with ID starting with 0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9 not found: ID does not exist" containerID="0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9" Oct 31 00:36:27.382644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.382493 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9"} err="failed to get container status \"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9\": rpc error: code = NotFound desc = could not find container \"0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9\": container with ID starting with 0ae4545f21ddf39fb0a53759509a3f4da7dee002ea402ee07bbd6da9664109f9 not found: ID does not exist" Oct 31 00:36:27.544064 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.543975 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:36:27.547042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:27.547018 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c2j7f"] Oct 31 00:36:28.021221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:28.021179 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:29.208627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:29.208592 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" path="/var/lib/kubelet/pods/daa66a9f-1f63-4b29-8ce1-7ae32d47a96e/volumes" Oct 31 00:36:33.225078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:33.225039 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.393250 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:33.393212 4471 scope.go:122] "RemoveContainer" containerID="90411060af95da23837ff198b04e7704bc7c866daba443adec8d6378c04922ac" Oct 31 00:36:34.283416 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:36:34.283372 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:36:38.098026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:38.097989 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:43.008630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:43.008590 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:44.331416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.331379 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:36:44.331940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.331925 4471 cpu_manager.go:414] "containerMap: removing container" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" Oct 31 00:36:44.331985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.331942 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" Oct 31 00:36:44.332068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.332057 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa66a9f-1f63-4b29-8ce1-7ae32d47a96e" containerName="istio-proxy" Oct 31 00:36:44.336947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.336927 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346015 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346063 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346138 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4l7d\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346197 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346250 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346279 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346339 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346384 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.346482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.346470 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.441474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.441433 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-412743-openshift-default-dockercfg-wqjw6\"" Oct 31 00:36:44.444295 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.444269 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:36:44.447660 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447638 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447683 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447703 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4l7d\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447721 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447762 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447786 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447818 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.447958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.447920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.448234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.448208 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.448319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.448282 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.448377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.448322 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.448434 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.448386 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.448730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.448704 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.449954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.449938 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.450271 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.450256 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.508964 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.508924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.509130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.509014 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4l7d\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\") pod \"e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:44.650155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:44.650111 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:45.074755 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:36:45.074724 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64cc4b4_f2a4_4639_8b58_3b0d7e5e467b.slice/crio-4283da88de66eeab3bc797fac0e649ee1b6a875f040737237ffaad92de0c747f WatchSource:0}: Error finding container 4283da88de66eeab3bc797fac0e649ee1b6a875f040737237ffaad92de0c747f: Status 404 returned error can't find the container with id 4283da88de66eeab3bc797fac0e649ee1b6a875f040737237ffaad92de0c747f Oct 31 00:36:45.076511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.076487 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:36:45.077048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.077017 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:45.077092 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.077081 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:45.077125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.077112 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:36:45.472368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.472333 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" event={"ID":"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b","Type":"ContainerStarted","Data":"7f833c10e0fd36c7620bdc4bb3d12461ba65e70704e6a50001f4d90dae66fd68"} Oct 31 00:36:45.472802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.472396 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" event={"ID":"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b","Type":"ContainerStarted","Data":"4283da88de66eeab3bc797fac0e649ee1b6a875f040737237ffaad92de0c747f"} Oct 31 00:36:45.597926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.597869 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podStartSLOduration=1.597853927 podStartE2EDuration="1.597853927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:45.596106208 +0000 UTC m=+1332.521375159" watchObservedRunningTime="2026-10-31 00:36:45.597853927 +0000 UTC m=+1332.523122972" Oct 31 00:36:45.650508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:45.650464 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:46.652452 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:46.652417 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:46.652898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:46.652470 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:47.650714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:47.650680 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:47.650903 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:47.650738 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:48.091945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:48.091863 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:48.651238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:48.651200 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:48.651448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:48.651267 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:49.650722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:49.650683 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:49.651124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:49.650744 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:50.651572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:50.651508 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:50.651964 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:50.651586 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:51.651614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:51.651564 4471 patch_prober.go:29] interesting pod/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:36:51.652015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:51.651647 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:36:52.673117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:52.673083 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:52.674216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:52.674070 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:52.675133 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:52.675111 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:36:53.215399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:53.215351 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:58.225663 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:36:58.225627 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:03.298261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:03.298225 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:04.280831 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:37:04.280793 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:37:08.311102 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:08.311055 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:13.282611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.282570 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:13.614510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.614426 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:37:13.618144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.618125 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.670405 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.670363 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-982ae3-openshift-default-dockercfg-cwtrb\"" Oct 31 00:37:13.705751 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.705719 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.705913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.705781 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.705913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.705878 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.705988 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.705916 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.706024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.705991 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.706059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.706032 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.706059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.706054 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj9h\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.706212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.706192 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.706285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.706270 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.722907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.722871 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:37:13.807229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807186 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807241 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807282 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807309 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807339 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807391 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807427 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807717 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807772 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj9h\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807858 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807880 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.807993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.807773 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.817560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.808622 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.817804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.808691 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.817888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.809819 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.819709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.819683 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.868705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.868633 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.868838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.868790 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj9h\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:13.931651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:13.931619 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:14.357001 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:14.356969 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892d7291_94c7_4d14_819c_e80df40bd347.slice/crio-b4018f9b05653954e3244012b23864891eed4a34606756863e72bca93fed7d7c WatchSource:0}: Error finding container b4018f9b05653954e3244012b23864891eed4a34606756863e72bca93fed7d7c: Status 404 returned error can't find the container with id b4018f9b05653954e3244012b23864891eed4a34606756863e72bca93fed7d7c Oct 31 00:37:14.359268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.359231 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:14.359386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.359300 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:14.359386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.359330 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:14.359486 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.359448 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:37:14.647384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.647346 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" event={"ID":"892d7291-94c7-4d14-819c-e80df40bd347","Type":"ContainerStarted","Data":"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632"} Oct 31 00:37:14.647580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.647400 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" event={"ID":"892d7291-94c7-4d14-819c-e80df40bd347","Type":"ContainerStarted","Data":"b4018f9b05653954e3244012b23864891eed4a34606756863e72bca93fed7d7c"} Oct 31 00:37:14.769809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.769755 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podStartSLOduration=1.769739293 podStartE2EDuration="1.769739293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:14.767512271 +0000 UTC m=+1361.692781222" watchObservedRunningTime="2026-10-31 00:37:14.769739293 +0000 UTC m=+1361.695008246" Oct 31 00:37:14.932012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:14.931928 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:15.934542 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:15.934495 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:15.934927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:15.934578 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:16.932464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:16.932427 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:16.932670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:16.932486 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:17.932442 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:17.932402 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:17.932852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:17.932456 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:18.293296 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:18.293190 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:18.932874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:18.932828 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:18.933375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:18.932905 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:19.932541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:19.932503 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:19.932754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:19.932597 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:20.932447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:20.932412 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 31 00:37:20.932882 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:20.932467 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 31 00:37:21.936687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:21.936652 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:21.937113 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:21.936923 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:21.937793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:21.937771 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:37:23.499018 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:23.498981 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:23.649929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:23.649883 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:25.574135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.574094 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq"] Oct 31 00:37:25.577924 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.577907 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.630789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.630756 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:37:25.679743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.679707 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.679743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.679746 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrkw\" (UniqueName: \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.679987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.679962 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.680482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.680465 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:37:25.682768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.682748 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq"] Oct 31 00:37:25.781610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.781571 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.781803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.781637 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.781803 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.781665 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrkw\" (UniqueName: \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.782459 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.782423 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.784011 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.783988 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.842902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.842826 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrkw\" (UniqueName: \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\") pod \"maas-api-revoke-sub-fdda52da-p79sq\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:25.891283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:25.891246 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:26.268301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:26.266139 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq"] Oct 31 00:37:26.733563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:26.733501 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" event={"ID":"704a490e-9a23-4aff-b3df-4d1afc7a79ae","Type":"ContainerStarted","Data":"5d7601661e8c3c2eb9c80f83131bdd16cfea46224b42ae906deb12244140e2a4"} Oct 31 00:37:26.733987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:26.733573 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" event={"ID":"704a490e-9a23-4aff-b3df-4d1afc7a79ae","Type":"ContainerStarted","Data":"7726533f3e285fca3525ae24a3778378663eba59e80e6e9918bb7fc55af9660f"} Oct 31 00:37:26.907619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:26.876580 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" podStartSLOduration=1.87655912 podStartE2EDuration="1.87655912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:26.872930226 +0000 UTC m=+1373.798199179" watchObservedRunningTime="2026-10-31 00:37:26.87655912 +0000 UTC m=+1373.801828075" Oct 31 00:37:27.748563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:27.746492 4471 generic.go:356] "Generic (PLEG): container finished" podID="704a490e-9a23-4aff-b3df-4d1afc7a79ae" containerID="5d7601661e8c3c2eb9c80f83131bdd16cfea46224b42ae906deb12244140e2a4" exitCode=0 Oct 31 00:37:27.748563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:27.746676 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" event={"ID":"704a490e-9a23-4aff-b3df-4d1afc7a79ae","Type":"ContainerDied","Data":"5d7601661e8c3c2eb9c80f83131bdd16cfea46224b42ae906deb12244140e2a4"} Oct 31 00:37:28.902062 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.902031 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:28.959682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.959640 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\") pod \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " Oct 31 00:37:28.959862 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.959717 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\") pod \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " Oct 31 00:37:28.959862 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.959791 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\" (UniqueName: \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\") pod \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\" (UID: \"704a490e-9a23-4aff-b3df-4d1afc7a79ae\") " Oct 31 00:37:28.960071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.960031 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca" pod "704a490e-9a23-4aff-b3df-4d1afc7a79ae" (UID: "704a490e-9a23-4aff-b3df-4d1afc7a79ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:28.960310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.960294 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:28.961779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.961757 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca" pod "704a490e-9a23-4aff-b3df-4d1afc7a79ae" (UID: "704a490e-9a23-4aff-b3df-4d1afc7a79ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:28.961925 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:28.961907 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw" pod "704a490e-9a23-4aff-b3df-4d1afc7a79ae" (UID: "704a490e-9a23-4aff-b3df-4d1afc7a79ae"). InnerVolumeSpecName "kube-api-access-5jrkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:29.061545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.061458 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704a490e-9a23-4aff-b3df-4d1afc7a79ae-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:29.061545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.061485 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrkw\" (UniqueName: \"kubernetes.io/projected/704a490e-9a23-4aff-b3df-4d1afc7a79ae-kube-api-access-5jrkw\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:29.168117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.168077 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:29.759746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.759702 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" event={"ID":"704a490e-9a23-4aff-b3df-4d1afc7a79ae","Type":"ContainerDied","Data":"7726533f3e285fca3525ae24a3778378663eba59e80e6e9918bb7fc55af9660f"} Oct 31 00:37:29.759746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.759727 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq" Oct 31 00:37:29.759997 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:29.759757 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7726533f3e285fca3525ae24a3778378663eba59e80e6e9918bb7fc55af9660f" Oct 31 00:37:34.281545 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:37:34.281490 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:37:34.285629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:34.285585 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:35.618627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:35.618568 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:38.778424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:38.778389 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:38.936638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:38.936603 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.308244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:39.308211 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:39.635148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:39.635041 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:40.158942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:40.158866 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:40.353556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:40.353491 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:40.510481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:40.510385 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:41.338280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:41.338231 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:41.537268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:41.537229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:43.771846 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:43.771768 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:44.498559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.498514 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:37:44.499065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.499047 4471 cpu_manager.go:414] "containerMap: removing container" podUID="704a490e-9a23-4aff-b3df-4d1afc7a79ae" containerName="revoke-keys" Oct 31 00:37:44.499128 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.499067 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704a490e-9a23-4aff-b3df-4d1afc7a79ae" containerName="revoke-keys" Oct 31 00:37:44.499178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.499169 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="704a490e-9a23-4aff-b3df-4d1afc7a79ae" containerName="revoke-keys" Oct 31 00:37:44.502580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.502556 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.621368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.621336 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:37:44.640301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.640268 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:44.670319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670289 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670339 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670476 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8cn\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670579 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670636 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670720 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670749 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670776 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.670898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.670877 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.772542 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.772428 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773118 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773269 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773391 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773516 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773489 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773620 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773795 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.773955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.773939 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774114 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8cn\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774122 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774636 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774359 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774833 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.774968 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.774956 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.775996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.775953 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.781287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.781265 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.801967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.801941 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:44.837022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.836996 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.837168 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.837116 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8cn\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\") pod \"e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:44.957039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:44.957009 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:45.116540 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.116444 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:45.133559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.133497 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:45.284204 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.284165 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:45.447273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.447241 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:45.545984 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:45.545954 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac398aec_72bf_4698_a1dd_83133739184b.slice/crio-0ae191189fb6e1af5957301a5375001229ed6051c6caf96cfce2c448d51c9b36 WatchSource:0}: Error finding container 0ae191189fb6e1af5957301a5375001229ed6051c6caf96cfce2c448d51c9b36: Status 404 returned error can't find the container with id 0ae191189fb6e1af5957301a5375001229ed6051c6caf96cfce2c448d51c9b36 Oct 31 00:37:45.547573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.547545 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:37:45.548730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.548697 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:45.548838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.548759 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:45.548838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.548805 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:45.849586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.849465 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" event={"ID":"ac398aec-72bf-4698-a1dd-83133739184b","Type":"ContainerStarted","Data":"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08"} Oct 31 00:37:45.849586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.849548 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" event={"ID":"ac398aec-72bf-4698-a1dd-83133739184b","Type":"ContainerStarted","Data":"0ae191189fb6e1af5957301a5375001229ed6051c6caf96cfce2c448d51c9b36"} Oct 31 00:37:45.974049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:45.973992 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podStartSLOduration=1.9739791869999999 podStartE2EDuration="1.973979187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:45.972584306 +0000 UTC m=+1392.897853255" watchObservedRunningTime="2026-10-31 00:37:45.973979187 +0000 UTC m=+1392.899248139" Oct 31 00:37:46.117645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:46.117569 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:46.119266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:46.119237 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:46.119400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:46.119289 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:47.117276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:47.117235 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:47.117781 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:47.117302 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:47.708554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:47.692288 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.838581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:47.838512 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:48.005757 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.005649 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:48.117434 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.117400 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:48.117850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.117454 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:48.654357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.654296 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:48.810877 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.810846 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:48.984133 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:48.984038 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:49.117427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:49.117391 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:49.117605 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:49.117452 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:49.137288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:49.137232 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.055980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:50.055937 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.116995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:50.116960 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:50.117205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:50.117015 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:50.219633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:50.219580 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:50.590881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:50.590847 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.117726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:51.117687 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:51.117905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:51.117751 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:51.145872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:51.145822 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.313855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:51.313824 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:52.116818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:52.116782 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:37:52.116996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:52.116840 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:37:53.121915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:53.121880 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:53.122302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:53.122248 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:53.123087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:53.123067 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:37:53.606232 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:53.606185 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.775438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:53.775404 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.013956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.013925 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:37:54.017685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.017664 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.074324 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.074281 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e7b593fa-dockercfg-4btlc\"" Oct 31 00:37:54.074949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.074921 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e7b593fa\"" Oct 31 00:37:54.077870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.077846 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:37:54.103551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.089361 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.174381 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.174345 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:37:54.178042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.178026 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.218103 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.218074 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28kc6\" (UniqueName: \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.218287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.218268 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jw9q\" (UniqueName: \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.218366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.218310 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.218366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.218338 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.235885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.235852 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:37:54.283345 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.283262 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.319255 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.319218 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jw9q\" (UniqueName: \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.319428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.319268 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.319523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.319502 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.319625 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.319585 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kc6\" (UniqueName: \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.320056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.320039 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.320185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.320170 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.381008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.380979 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kc6\" (UniqueName: \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\") pod \"payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.381147 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.380979 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jw9q\" (UniqueName: \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\") pod \"payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.437191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.437161 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.491215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.491189 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:54.631196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.631166 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:54.862645 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:54.862612 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672541b4_3959_4f5f_bf63_faf9b3753a66.slice/crio-3db7499c3c417426a7fe7dfc4c3ace75c8c358abe38c487339a3d266b15ab4e9 WatchSource:0}: Error finding container 3db7499c3c417426a7fe7dfc4c3ace75c8c358abe38c487339a3d266b15ab4e9: Status 404 returned error can't find the container with id 3db7499c3c417426a7fe7dfc4c3ace75c8c358abe38c487339a3d266b15ab4e9 Oct 31 00:37:54.864961 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.864936 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:37:54.930301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.930268 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" event={"ID":"672541b4-3959-4f5f-bf63-faf9b3753a66","Type":"ContainerStarted","Data":"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a"} Oct 31 00:37:54.930413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:54.930330 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" event={"ID":"672541b4-3959-4f5f-bf63-faf9b3753a66","Type":"ContainerStarted","Data":"3db7499c3c417426a7fe7dfc4c3ace75c8c358abe38c487339a3d266b15ab4e9"} Oct 31 00:37:54.998813 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:54.998782 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209a528c_be2f_4d6e_8843_02458d09e26d.slice/crio-d82319dce68ca5a640bd2529e3fcca49c137e59b71819b494a2dfbfa2e4a4f7a WatchSource:0}: Error finding container d82319dce68ca5a640bd2529e3fcca49c137e59b71819b494a2dfbfa2e4a4f7a: Status 404 returned error can't find the container with id d82319dce68ca5a640bd2529e3fcca49c137e59b71819b494a2dfbfa2e4a4f7a Oct 31 00:37:55.000947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.000921 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:37:55.046992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.046948 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" podStartSLOduration=1.046934938 podStartE2EDuration="1.046934938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:55.04622131 +0000 UTC m=+1401.971490264" watchObservedRunningTime="2026-10-31 00:37:55.046934938 +0000 UTC m=+1401.972203887" Oct 31 00:37:55.937208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.937167 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" event={"ID":"209a528c-be2f-4d6e-8843-02458d09e26d","Type":"ContainerStarted","Data":"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72"} Oct 31 00:37:55.937676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.937448 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:37:55.937676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.937498 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:37:55.937676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:55.937524 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" event={"ID":"209a528c-be2f-4d6e-8843-02458d09e26d","Type":"ContainerStarted","Data":"d82319dce68ca5a640bd2529e3fcca49c137e59b71819b494a2dfbfa2e4a4f7a"} Oct 31 00:37:56.054876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:56.054821 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" podStartSLOduration=3.05480774 podStartE2EDuration="3.05480774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:56.052256034 +0000 UTC m=+1402.977524988" watchObservedRunningTime="2026-10-31 00:37:56.05480774 +0000 UTC m=+1402.980076693" Oct 31 00:37:56.184651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:56.184621 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:56.914711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:56.914678 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.060907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.060876 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:57.764096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.764053 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:37:57.768070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.768043 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.815193 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.815159 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:37:57.819289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.819265 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858233 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858274 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.858446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858298 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858455 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858557 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858591 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.858762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858612 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.858762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858719 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858759 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.858867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858827 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.858973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858868 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6fsv\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858898 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.858973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.858969 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.859125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.859018 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.859125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.859100 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.859226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.859158 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.859226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.859183 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk84l\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.859226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.859220 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.866371 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.866342 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:37:57.882259 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.882228 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:37:57.960385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960355 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.960574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960406 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.960574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960449 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk84l\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.960574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960480 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.960574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960540 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.960574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960572 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960608 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960647 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960687 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960713 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960739 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960771 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960809 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960869 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960909 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6fsv\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960914 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960945 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961079 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961166 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961440 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961493 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961712 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961776 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.961813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.960994 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.962180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.961992 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.962180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.962023 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.962431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.962389 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.963456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.963433 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:57.963736 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.963715 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.963815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.963745 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:57.964004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:57.963985 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:58.022891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.022819 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:58.023031 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.022996 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk84l\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:58.023265 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.023246 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6fsv\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:58.030282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.030260 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\") pod \"e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:58.081584 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.081557 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:58.134195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.133386 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:58.273309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.273226 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.504200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.504165 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.513284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.513253 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:37:58.515134 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:58.515101 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6e4f0d_30df_4293_a749_93f0de3fed63.slice/crio-b25c3395a6f5d2c2e8ea3acd8237a6ca069ba7a40919ef1f636cbc84e02e6993 WatchSource:0}: Error finding container b25c3395a6f5d2c2e8ea3acd8237a6ca069ba7a40919ef1f636cbc84e02e6993: Status 404 returned error can't find the container with id b25c3395a6f5d2c2e8ea3acd8237a6ca069ba7a40919ef1f636cbc84e02e6993 Oct 31 00:37:58.518015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.517981 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.518098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.518051 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.518098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.518077 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.604194 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:37:58.604163 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e3de49_d2d9_4f71_be1a_de0effb425c5.slice/crio-64963f7ea5b5a41eaf6fc74ae2fbd92c51312c9594f2422a7fe527d56147f7ee WatchSource:0}: Error finding container 64963f7ea5b5a41eaf6fc74ae2fbd92c51312c9594f2422a7fe527d56147f7ee: Status 404 returned error can't find the container with id 64963f7ea5b5a41eaf6fc74ae2fbd92c51312c9594f2422a7fe527d56147f7ee Oct 31 00:37:58.606042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.606019 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:37:58.606809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.606778 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.606884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.606842 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.606884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.606872 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:37:58.698967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.698932 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:58.958942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.958896 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" event={"ID":"a6e3de49-d2d9-4f71-be1a-de0effb425c5","Type":"ContainerStarted","Data":"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152"} Oct 31 00:37:58.959148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.958967 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" event={"ID":"a6e3de49-d2d9-4f71-be1a-de0effb425c5","Type":"ContainerStarted","Data":"64963f7ea5b5a41eaf6fc74ae2fbd92c51312c9594f2422a7fe527d56147f7ee"} Oct 31 00:37:58.961106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.961068 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" event={"ID":"ef6e4f0d-30df-4293-a749-93f0de3fed63","Type":"ContainerStarted","Data":"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4"} Oct 31 00:37:58.961239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:58.961129 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" event={"ID":"ef6e4f0d-30df-4293-a749-93f0de3fed63","Type":"ContainerStarted","Data":"b25c3395a6f5d2c2e8ea3acd8237a6ca069ba7a40919ef1f636cbc84e02e6993"} Oct 31 00:37:59.081719 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.081670 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podStartSLOduration=2.081655556 podStartE2EDuration="2.081655556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.078932539 +0000 UTC m=+1406.004201493" watchObservedRunningTime="2026-10-31 00:37:59.081655556 +0000 UTC m=+1406.006924508" Oct 31 00:37:59.082124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.081922 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:37:59.083176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.083154 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:37:59.083273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.083198 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:37:59.135602 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.135567 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:37:59.136656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.136632 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:37:59.136771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.136684 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:37:59.200970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:37:59.200916 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podStartSLOduration=2.200902102 podStartE2EDuration="2.200902102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:59.198618306 +0000 UTC m=+1406.123887259" watchObservedRunningTime="2026-10-31 00:37:59.200902102 +0000 UTC m=+1406.126171054" Oct 31 00:38:00.083556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.082942 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:00.083556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.083011 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:00.135364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.135325 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:00.135567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.135391 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:00.702555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.681329 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:00.855584 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:00.855347 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.013611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:01.012981 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.093551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:01.092998 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:01.093551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:01.093065 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:01.135191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:01.135155 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:01.135343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:01.135211 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:02.085555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:02.082982 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:02.085555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:02.083047 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:02.102555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:02.095120 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.135727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:02.135691 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:02.135902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:02.135744 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:03.091713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:03.091670 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:03.092030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:03.091991 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:03.141146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:03.141108 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:03.141778 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:03.141743 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:04.006560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:03.996662 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.094551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.094099 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:04.094551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.094182 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:04.097559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.096684 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.145986 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.145948 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:04.146573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.146519 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:04.323098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.322968 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.381555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.378656 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 31 00:38:04.457578 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.444165 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.683584 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:04.679720 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:38:04.761556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.759634 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:04.845555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:04.833814 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:05.033771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.032636 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:05.094679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.094640 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 31 00:38:05.094970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.094935 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 31 00:38:05.142551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.135943 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 00:38:05.142551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.136013 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 00:38:05.588409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.588375 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:05.753748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:05.753715 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:06.040481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.040445 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:06.480595 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.480556 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:38:06.481150 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.481121 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:38:06.482410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.482388 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:38:06.605492 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.605458 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:38:06.606111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.605906 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:38:06.606764 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.606747 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:38:06.946000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.945973 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:38:06.946197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:06.946041 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:38:07.736025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:07.735990 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:11.623897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:11.623861 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.028374 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:12.028337 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.202399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:12.202368 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.357005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:12.356896 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.620846 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:12.620755 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:12.780290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:12.780252 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:13.907845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:13.907808 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:38:13.913007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:13.912987 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:13.992200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:13.992166 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:38:14.041292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041256 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041295 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041543 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041543 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041497 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041613 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041649 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041678 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97f8\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041762 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.041815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.041800 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.093706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.093672 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:38:14.097635 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.097612 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.142648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.142611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97f8\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.142826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.142670 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.142826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.142713 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.142927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.142863 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.142927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.142911 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143055 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143121 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143189 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143246 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143331 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143305 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143648 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143692 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.143898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.143842 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.144303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.144278 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.145451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.145428 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.145805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.145784 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.187982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.187896 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:38:14.207727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.207689 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97f8\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.207900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.207696 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\") pod \"e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.227395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.227355 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:14.244185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244155 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244224 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244331 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75nt\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244411 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244474 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244665 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244647 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244688 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244721 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.244851 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.244810 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346106 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346216 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346345 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346401 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346463 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346500 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346643 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346664 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346648 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346761 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.346816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.346808 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75nt\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.347132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.347106 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.347546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.347493 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.347546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.347501 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.349130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.349106 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.349290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.349274 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.407808 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.407773 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.407967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.407900 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75nt\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.410718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.410702 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:14.654224 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:14.654187 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7184561_f832_47f3_aed5_5e6a44f2d18f.slice/crio-24b836bf687849f19cd31f5870281b6c9bf684032ea11f7810728f54a5ecef97 WatchSource:0}: Error finding container 24b836bf687849f19cd31f5870281b6c9bf684032ea11f7810728f54a5ecef97: Status 404 returned error can't find the container with id 24b836bf687849f19cd31f5870281b6c9bf684032ea11f7810728f54a5ecef97 Oct 31 00:38:14.656451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.656425 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:38:14.656876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.656844 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:14.656933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.656907 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:14.656967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.656939 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:14.786639 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:14.786603 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c616615_724c_46dd_abf7_c5adbe371156.slice/crio-5fcb531a40c9368130f6e9035748c365d9ed8f08b0f935e40cf997f026882823 WatchSource:0}: Error finding container 5fcb531a40c9368130f6e9035748c365d9ed8f08b0f935e40cf997f026882823: Status 404 returned error can't find the container with id 5fcb531a40c9368130f6e9035748c365d9ed8f08b0f935e40cf997f026882823 Oct 31 00:38:14.788306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.788280 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:38:14.789626 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.789596 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:14.789699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.789662 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:14.789699 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:14.789688 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:15.193965 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.193921 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" event={"ID":"9c616615-724c-46dd-abf7-c5adbe371156","Type":"ContainerStarted","Data":"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6"} Oct 31 00:38:15.194566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.194519 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" event={"ID":"9c616615-724c-46dd-abf7-c5adbe371156","Type":"ContainerStarted","Data":"5fcb531a40c9368130f6e9035748c365d9ed8f08b0f935e40cf997f026882823"} Oct 31 00:38:15.195945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.195920 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" event={"ID":"d7184561-f832-47f3-aed5-5e6a44f2d18f","Type":"ContainerStarted","Data":"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091"} Oct 31 00:38:15.196056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.195967 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" event={"ID":"d7184561-f832-47f3-aed5-5e6a44f2d18f","Type":"ContainerStarted","Data":"24b836bf687849f19cd31f5870281b6c9bf684032ea11f7810728f54a5ecef97"} Oct 31 00:38:15.228096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.228066 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:15.229948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.229921 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:15.230070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.229976 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:15.318899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.318849 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podStartSLOduration=1.318834629 podStartE2EDuration="1.318834629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:15.317237409 +0000 UTC m=+1422.242506360" watchObservedRunningTime="2026-10-31 00:38:15.318834629 +0000 UTC m=+1422.244103579" Oct 31 00:38:15.411573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.411512 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:15.413264 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.413234 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:15.413393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.413285 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:15.443277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:15.443227 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podStartSLOduration=2.443214564 podStartE2EDuration="2.443214564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:15.442442352 +0000 UTC m=+1422.367711505" watchObservedRunningTime="2026-10-31 00:38:15.443214564 +0000 UTC m=+1422.368483574" Oct 31 00:38:16.227971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:16.227937 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:16.228370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:16.227984 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:16.411238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:16.411191 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:16.411450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:16.411266 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:17.228600 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:17.228569 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:17.229084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:17.228618 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:17.411689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:17.411263 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:17.411689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:17.411332 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:18.088553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.088052 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:18.227967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.227930 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:18.228178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.227992 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:18.411207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.411169 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:18.411672 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.411231 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:18.450873 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:18.450840 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:19.228597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.228504 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:19.228597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.228580 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:19.411763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.411418 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:19.411763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.411480 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:19.531644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.531558 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:19.685566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:19.685081 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.228647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:20.228608 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:20.228847 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:20.228676 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:20.411910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:20.411877 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:20.412392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:20.412355 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:21.228415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:21.228370 4471 patch_prober.go:29] interesting pod/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.228628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:21.228454 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:38:21.412179 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:21.412134 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:38:21.412620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:21.412208 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:38:22.232687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.232655 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:22.233144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.233127 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:22.233760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.233739 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:38:22.416069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.416036 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:22.416556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.416391 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:22.417114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:22.417095 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:38:23.519610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:23.519569 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:23.742789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:23.742757 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.204928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.204891 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:38:28.209188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.209169 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.305245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.305209 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.305436 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.305284 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.305669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.305638 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmkf\" (UniqueName: \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.305875 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.305850 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.312992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.312964 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-26fb7237-serving-cert\"" Oct 31 00:38:28.318866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.318839 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:38:28.407131 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.407096 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.407298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.407226 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmkf\" (UniqueName: \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.407298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.407248 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.407367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.407324 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.407448 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:28.407436 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-serving-cert: secret "maas-api-e2e-migrate-26fb7237-serving-cert" not found Oct 31 00:38:28.407514 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:28.407505 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls podName:38c05a9a-f25e-4ea6-8553-f995fce69e67 nodeName:}" failed. No retries permitted until 2026-10-31 00:38:28.907489697 +0000 UTC m=+1435.832758628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls") pod "maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" (UID: "38c05a9a-f25e-4ea6-8553-f995fce69e67") : secret "maas-api-e2e-migrate-26fb7237-serving-cert" not found Oct 31 00:38:28.408007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.407985 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.409464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.409446 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.424240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.424213 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.468407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.468338 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmkf\" (UniqueName: \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.584617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.584576 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.836849 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.836764 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:28.913765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.913728 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:28.916219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:28.916195 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") pod \"maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:29.123801 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:29.123711 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:29.494021 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:29.493992 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c05a9a_f25e_4ea6_8553_f995fce69e67.slice/crio-857746feb637a3cf219d7d9b07db8814dc5470102083ef90492dcc7e51fedacd WatchSource:0}: Error finding container 857746feb637a3cf219d7d9b07db8814dc5470102083ef90492dcc7e51fedacd: Status 404 returned error can't find the container with id 857746feb637a3cf219d7d9b07db8814dc5470102083ef90492dcc7e51fedacd Oct 31 00:38:29.495613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:29.495591 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:38:29.498409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:29.498379 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:38:30.296770 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:30.296730 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" event={"ID":"38c05a9a-f25e-4ea6-8553-f995fce69e67","Type":"ContainerStarted","Data":"857746feb637a3cf219d7d9b07db8814dc5470102083ef90492dcc7e51fedacd"} Oct 31 00:38:31.304343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:31.304303 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" event={"ID":"38c05a9a-f25e-4ea6-8553-f995fce69e67","Type":"ContainerStarted","Data":"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e"} Oct 31 00:38:31.304777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:31.304371 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:31.424038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:31.423969 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" podStartSLOduration=2.488591672 podStartE2EDuration="3.423950304s" totalImagesPullingTime="935.358632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:31.422276003 +0000 UTC m=+1438.347544960" watchObservedRunningTime="2026-10-31 00:38:31.423950304 +0000 UTC m=+1438.349219257" Oct 31 00:38:33.369775 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:33.369732 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:34.281509 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:34.281473 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:38:37.316963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.316934 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:38:37.754850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.754820 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:38:37.760033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.760012 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.818228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.818194 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fxh6q\"" Oct 31 00:38:37.823574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823543 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.823735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823601 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.823735 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823717 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.823832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823781 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.823938 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823908 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.823988 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.823946 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5r8d\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.824038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.824009 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.824092 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.824055 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.824156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.824139 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.870717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.870685 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:38:37.925025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.924993 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925057 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925107 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925153 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925214 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925287 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925312 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5r8d\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925490 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925383 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925490 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925434 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925700 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.925883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925859 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.926026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925948 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.926026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.925953 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.926026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.926000 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.927686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.927666 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.927788 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.927715 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.986855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.986825 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5r8d\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:37.987042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:37.986882 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-wspmm\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:38.044753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.044679 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:38:38.049910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.049884 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.073420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.073391 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:38.104506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.104476 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e7b593fa-serving-cert\"" Oct 31 00:38:38.107290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.107256 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:38:38.129040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.128593 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.129040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.128673 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.129040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.128805 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzvp\" (UniqueName: \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.129264 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.129057 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.230229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.230196 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.230402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.230312 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.230402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.230345 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.230509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.230425 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzvp\" (UniqueName: \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.230583 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:38.230568 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-serving-cert: secret "maas-api-e2e-ait-e7b593fa-serving-cert" not found Oct 31 00:38:38.230662 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:38.230650 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls podName:b8e07179-f097-43c7-a25e-e9a766dde88f nodeName:}" failed. No retries permitted until 2026-10-31 00:38:38.730625661 +0000 UTC m=+1445.655894606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls") pod "maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" (UID: "b8e07179-f097-43c7-a25e-e9a766dde88f") : secret "maas-api-e2e-ait-e7b593fa-serving-cert" not found Oct 31 00:38:38.230999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.230982 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.232597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.232579 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.245458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.245431 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.291963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.291936 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzvp\" (UniqueName: \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.395893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.395861 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.496654 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:38.496607 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2963405d_002b_4406_9002_27948db6bcb8.slice/crio-75bae741dbcc2e42cc87005504d01b6e947c338c2411243b8b8b7fa756f528cd WatchSource:0}: Error finding container 75bae741dbcc2e42cc87005504d01b6e947c338c2411243b8b8b7fa756f528cd: Status 404 returned error can't find the container with id 75bae741dbcc2e42cc87005504d01b6e947c338c2411243b8b8b7fa756f528cd Oct 31 00:38:38.502016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.501958 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:38.503750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.502054 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:38.503921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.503802 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:38:38.504027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.504007 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:38:38.553454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.553425 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:38.737665 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.737578 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.739932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.739906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") pod \"maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:38.964328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:38.964294 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:39.352662 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:39.352632 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e07179_f097_43c7_a25e_e9a766dde88f.slice/crio-5c2000f62ec4553d19ccde32fca911a91a12d879dd88e5e308a54ecefba8b769 WatchSource:0}: Error finding container 5c2000f62ec4553d19ccde32fca911a91a12d879dd88e5e308a54ecefba8b769: Status 404 returned error can't find the container with id 5c2000f62ec4553d19ccde32fca911a91a12d879dd88e5e308a54ecefba8b769 Oct 31 00:38:39.354002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:39.353976 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" event={"ID":"2963405d-002b-4406-9002-27948db6bcb8","Type":"ContainerStarted","Data":"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab"} Oct 31 00:38:39.354105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:39.354039 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" event={"ID":"2963405d-002b-4406-9002-27948db6bcb8","Type":"ContainerStarted","Data":"75bae741dbcc2e42cc87005504d01b6e947c338c2411243b8b8b7fa756f528cd"} Oct 31 00:38:39.354793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:39.354768 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:38:39.476281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:39.476229 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podStartSLOduration=2.476215105 podStartE2EDuration="2.476215105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:39.474120321 +0000 UTC m=+1446.399389275" watchObservedRunningTime="2026-10-31 00:38:39.476215105 +0000 UTC m=+1446.401484058" Oct 31 00:38:40.081782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:40.081728 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:40.083494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:40.083463 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:40.083660 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:40.083570 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:40.386553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:40.378210 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" event={"ID":"b8e07179-f097-43c7-a25e-e9a766dde88f","Type":"ContainerStarted","Data":"5c2000f62ec4553d19ccde32fca911a91a12d879dd88e5e308a54ecefba8b769"} Oct 31 00:38:41.076974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:41.076935 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:41.077610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:41.077575 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:41.388643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:41.388504 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" event={"ID":"b8e07179-f097-43c7-a25e-e9a766dde88f","Type":"ContainerStarted","Data":"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d"} Oct 31 00:38:41.388643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:41.388611 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:41.508476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:41.508422 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" podStartSLOduration=3.525712376 podStartE2EDuration="4.508406895s" totalImagesPullingTime="982.694519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:41.505281315 +0000 UTC m=+1448.430550268" watchObservedRunningTime="2026-10-31 00:38:41.508406895 +0000 UTC m=+1448.433675910" Oct 31 00:38:42.081583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:42.078947 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:42.081583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:42.079034 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:43.084581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:43.082091 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:43.084581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:43.082143 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:43.482304 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:43.482263 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:44.081033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:44.080987 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:44.081461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:44.081426 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:45.074520 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:45.074484 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:45.074951 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:45.074580 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:46.087608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:46.076926 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:38:46.087608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:46.076988 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:38:47.423758 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:47.423726 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:38:48.119234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.118733 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:38:48.119234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.118826 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:38:48.359668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.359631 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:48.360514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.360484 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:48.361274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.361246 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:38:48.854385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.854353 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:38:48.860292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.860260 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:48.914354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.913794 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49bb6b1a-serving-cert\"" Oct 31 00:38:48.917372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.917345 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:38:48.995115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.995054 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:48.995308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.995177 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:48.995308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.995269 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:48.995422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:48.995330 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948t2\" (UniqueName: \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.062655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.062614 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.096852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.096807 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.097053 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.096917 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.097053 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:49.096958 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-serving-cert: secret "maas-api-e2e-ait-49bb6b1a-serving-cert" not found Oct 31 00:38:49.097053 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.096985 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948t2\" (UniqueName: \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.097053 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:38:49.097032 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls podName:d296dc07-39f1-46ac-9433-734ad78ffcfe nodeName:}" failed. No retries permitted until 2026-10-31 00:38:49.597010926 +0000 UTC m=+1456.522279883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls") pod "maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" (UID: "d296dc07-39f1-46ac-9433-734ad78ffcfe") : secret "maas-api-e2e-ait-49bb6b1a-serving-cert" not found Oct 31 00:38:49.097309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.097200 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.098057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.098036 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.100356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.100331 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.165555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.161293 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948t2\" (UniqueName: \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.259127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.259087 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.409170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.409137 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.603784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.603683 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.606871 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.606843 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") pod \"maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:49.788281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:49.788250 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:50.173168 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:50.173134 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:38:50.174820 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:38:50.174788 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd296dc07_39f1_46ac_9433_734ad78ffcfe.slice/crio-4dce241d8868b1a508fa7fbcc9df03ee33da93c6b120097064e73538235af828 WatchSource:0}: Error finding container 4dce241d8868b1a508fa7fbcc9df03ee33da93c6b120097064e73538235af828: Status 404 returned error can't find the container with id 4dce241d8868b1a508fa7fbcc9df03ee33da93c6b120097064e73538235af828 Oct 31 00:38:50.484017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:50.483922 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" event={"ID":"d296dc07-39f1-46ac-9433-734ad78ffcfe","Type":"ContainerStarted","Data":"4dce241d8868b1a508fa7fbcc9df03ee33da93c6b120097064e73538235af828"} Oct 31 00:38:51.493448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:51.493409 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" event={"ID":"d296dc07-39f1-46ac-9433-734ad78ffcfe","Type":"ContainerStarted","Data":"425e77054cc43bb63e7ede8e941fbb244f2604a8bcd5b93125da948346402257"} Oct 31 00:38:51.493896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:51.493472 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:51.646896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:51.646834 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" podStartSLOduration=2.684625744 podStartE2EDuration="3.646818857s" totalImagesPullingTime="962.193113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:51.64490548 +0000 UTC m=+1458.570174451" watchObservedRunningTime="2026-10-31 00:38:51.646818857 +0000 UTC m=+1458.572087809" Oct 31 00:38:54.397936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:54.397877 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:57.504694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:57.504659 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:38:59.416674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:38:59.416640 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.280696 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:39:04.280646 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:39:05.892415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:05.892381 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:08.004796 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.004762 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2"] Oct 31 00:39:08.010289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.010266 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.117605 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.117567 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2"] Oct 31 00:39:08.146234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146196 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146252 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnnt\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-kube-api-access-slnnt\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146399 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146490 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146588 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146657 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146713 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5964ec3-db73-48d0-aba5-6ba343813cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146751 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.146779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.146770 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.247747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.247708 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.247747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.247750 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.247963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.247878 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.247963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.247940 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnnt\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-kube-api-access-slnnt\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248019 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248172 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248136 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248135 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248283 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248325 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248423 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248449 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5964ec3-db73-48d0-aba5-6ba343813cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248686 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.248837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.248771 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.249151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.249132 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5964ec3-db73-48d0-aba5-6ba343813cc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.250215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.250195 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.250362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.250343 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.310037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.309950 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.310201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.310117 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnnt\" (UniqueName: \"kubernetes.io/projected/f5964ec3-db73-48d0-aba5-6ba343813cc1-kube-api-access-slnnt\") pod \"maas-default-gateway-openshift-default-6886b96f9-h4vr2\" (UID: \"f5964ec3-db73-48d0-aba5-6ba343813cc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.323644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.323604 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:08.747318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.747280 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2"] Oct 31 00:39:08.776703 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:39:08.776672 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5964ec3_db73_48d0_aba5_6ba343813cc1.slice/crio-810681fc2f143bbe0f914f1dc2f7f08010f5f10e7b63ec1b8fe8b47476755ed9 WatchSource:0}: Error finding container 810681fc2f143bbe0f914f1dc2f7f08010f5f10e7b63ec1b8fe8b47476755ed9: Status 404 returned error can't find the container with id 810681fc2f143bbe0f914f1dc2f7f08010f5f10e7b63ec1b8fe8b47476755ed9 Oct 31 00:39:08.779223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.779189 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:08.779325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.779259 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:08.779325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:08.779287 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:09.471667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:09.471628 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.604454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:09.604418 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" event={"ID":"f5964ec3-db73-48d0-aba5-6ba343813cc1","Type":"ContainerStarted","Data":"28c3456ad8ea691203476c9f72bf321a02ad6002df4c41228deac5190c0453de"} Oct 31 00:39:09.604661 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:09.604468 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" event={"ID":"f5964ec3-db73-48d0-aba5-6ba343813cc1","Type":"ContainerStarted","Data":"810681fc2f143bbe0f914f1dc2f7f08010f5f10e7b63ec1b8fe8b47476755ed9"} Oct 31 00:39:09.727432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:09.727329 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podStartSLOduration=2.72731226 podStartE2EDuration="2.72731226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:09.725636497 +0000 UTC m=+1476.650905449" watchObservedRunningTime="2026-10-31 00:39:09.72731226 +0000 UTC m=+1476.652581211" Oct 31 00:39:10.323954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:10.323920 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:10.325771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:10.325746 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:10.325887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:10.325794 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:11.323912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:11.323868 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:11.324316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:11.323925 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:12.324357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:12.324320 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:12.324899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:12.324383 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:13.324258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:13.324166 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:13.324258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:13.324231 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:13.600738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:13.600655 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:13.745939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:13.745907 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:14.324401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.324364 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:14.324612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.324419 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:14.481739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.481697 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:14.497775 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.497744 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:39:14.502947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.502927 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544256 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544223 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544256 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544260 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjz9h\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544366 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544486 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544511 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544566 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544607 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544782 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.544920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.544821 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.647653 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.647726 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.647770 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.647846 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.647973 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.648018 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.648094 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.648125 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjz9h\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.648186 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.649276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.648889 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.653033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.649947 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.653033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.650278 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.653033 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.651212 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.656019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.655990 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.666254 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.666228 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.667334 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.667306 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.704499 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.704459 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:39:14.710568 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.710522 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.711366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.711294 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjz9h\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\") pod \"e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:14.817235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:14.817203 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:15.304912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.304871 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:39:15.324494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.324457 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-h4vr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:39:15.324641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.324553 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" podUID="f5964ec3-db73-48d0-aba5-6ba343813cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 00:39:15.324914 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:39:15.324689 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fd9edd_18e7_4e0e_89ce_c8570fd907c5.slice/crio-d25d8f4cb5280ed034cd26e6632fa9a1764ec93962b55a3c646537ae0e5befcb WatchSource:0}: Error finding container d25d8f4cb5280ed034cd26e6632fa9a1764ec93962b55a3c646537ae0e5befcb: Status 404 returned error can't find the container with id d25d8f4cb5280ed034cd26e6632fa9a1764ec93962b55a3c646537ae0e5befcb Oct 31 00:39:15.327518 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.327483 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:15.327629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.327602 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:15.327676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.327661 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:39:15.646032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.645997 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" event={"ID":"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5","Type":"ContainerStarted","Data":"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99"} Oct 31 00:39:15.646195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.646069 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" event={"ID":"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5","Type":"ContainerStarted","Data":"d25d8f4cb5280ed034cd26e6632fa9a1764ec93962b55a3c646537ae0e5befcb"} Oct 31 00:39:15.778656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.778586 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podStartSLOduration=1.7785673260000001 podStartE2EDuration="1.778567326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:15.772386884 +0000 UTC m=+1482.697655851" watchObservedRunningTime="2026-10-31 00:39:15.778567326 +0000 UTC m=+1482.703836280" Oct 31 00:39:15.818181 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:15.818148 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:16.819498 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:16.819458 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:16.819967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:16.819554 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:17.324604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:17.324449 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:17.325816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:17.324970 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:17.326182 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:17.326155 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-h4vr2" Oct 31 00:39:17.817854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:17.817812 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:17.818042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:17.817875 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:18.818649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:18.818606 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:18.819156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:18.818664 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:19.683168 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:19.683132 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:19.818118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:19.818077 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:19.818359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:19.818133 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:20.817628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:20.817590 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:20.818049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:20.817646 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:21.818340 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:21.818296 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 00:39:21.818874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:21.818358 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 00:39:22.947889 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:22.947856 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:22.948453 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:22.948433 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:22.949343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:22.949324 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:39:24.147995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:24.147956 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:25.118196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:25.118164 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:29.919266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:29.919231 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:34.285814 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:39:34.285777 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:39:35.709183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:35.709148 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:41.372884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:41.372848 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:45.357610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:45.357580 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:45.514388 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:45.514324 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.586722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:46.586682 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:52.194727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:52.194687 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:53.301275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:53.301238 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:57.739992 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:57.739957 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:59.256300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:59.256259 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:39:59.366558 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:59.365523 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:39:59.610167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:59.610023 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:59.781918 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:39:59.781863 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:00.437435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:00.437402 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:02.892207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:02.892153 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:04.257575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.257518 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" podUID="672541b4-3959-4f5f-bf63-faf9b3753a66" containerName="payload-processing" containerID="cri-o://fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a" gracePeriod=25 Oct 31 00:40:04.283153 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:04.283113 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:40:04.366692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.366650 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" podUID="209a528c-be2f-4d6e-8843-02458d09e26d" containerName="payload-pre-processing" containerID="cri-o://9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72" gracePeriod=25 Oct 31 00:40:04.539857 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.539830 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:40:04.624446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.624414 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:40:04.671840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.671802 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\") pod \"672541b4-3959-4f5f-bf63-faf9b3753a66\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " Oct 31 00:40:04.672038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.671973 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\" (UniqueName: \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\") pod \"672541b4-3959-4f5f-bf63-faf9b3753a66\" (UID: \"672541b4-3959-4f5f-bf63-faf9b3753a66\") " Oct 31 00:40:04.672242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.672216 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume" pod "672541b4-3959-4f5f-bf63-faf9b3753a66" (UID: "672541b4-3959-4f5f-bf63-faf9b3753a66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:04.672598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.672572 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672541b4-3959-4f5f-bf63-faf9b3753a66-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:04.674139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.674117 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q" pod "672541b4-3959-4f5f-bf63-faf9b3753a66" (UID: "672541b4-3959-4f5f-bf63-faf9b3753a66"). InnerVolumeSpecName "kube-api-access-6jw9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:04.773385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.773295 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\" (UniqueName: \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\") pod \"209a528c-be2f-4d6e-8843-02458d09e26d\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " Oct 31 00:40:04.773385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.773337 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\") pod \"209a528c-be2f-4d6e-8843-02458d09e26d\" (UID: \"209a528c-be2f-4d6e-8843-02458d09e26d\") " Oct 31 00:40:04.774005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.773967 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume" pod "209a528c-be2f-4d6e-8843-02458d09e26d" (UID: "209a528c-be2f-4d6e-8843-02458d09e26d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:04.774281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.774265 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jw9q\" (UniqueName: \"kubernetes.io/projected/672541b4-3959-4f5f-bf63-faf9b3753a66-kube-api-access-6jw9q\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:04.774281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.774282 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209a528c-be2f-4d6e-8843-02458d09e26d-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:04.776042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.776014 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6" pod "209a528c-be2f-4d6e-8843-02458d09e26d" (UID: "209a528c-be2f-4d6e-8843-02458d09e26d"). InnerVolumeSpecName "kube-api-access-28kc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:04.875256 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.875222 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28kc6\" (UniqueName: \"kubernetes.io/projected/209a528c-be2f-4d6e-8843-02458d09e26d-kube-api-access-28kc6\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:04.987274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.987238 4471 generic.go:356] "Generic (PLEG): container finished" podID="672541b4-3959-4f5f-bf63-faf9b3753a66" containerID="fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a" exitCode=0 Oct 31 00:40:04.987465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.987293 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" Oct 31 00:40:04.987465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.987333 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" event={"ID":"672541b4-3959-4f5f-bf63-faf9b3753a66","Type":"ContainerDied","Data":"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a"} Oct 31 00:40:04.987465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.987395 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc" event={"ID":"672541b4-3959-4f5f-bf63-faf9b3753a66","Type":"ContainerDied","Data":"3db7499c3c417426a7fe7dfc4c3ace75c8c358abe38c487339a3d266b15ab4e9"} Oct 31 00:40:04.987465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.987419 4471 scope.go:122] "RemoveContainer" containerID="fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a" Oct 31 00:40:04.988828 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.988809 4471 generic.go:356] "Generic (PLEG): container finished" podID="209a528c-be2f-4d6e-8843-02458d09e26d" containerID="9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72" exitCode=0 Oct 31 00:40:04.988945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.988841 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" Oct 31 00:40:04.988945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.988846 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" event={"ID":"209a528c-be2f-4d6e-8843-02458d09e26d","Type":"ContainerDied","Data":"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72"} Oct 31 00:40:04.988945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.988879 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm" event={"ID":"209a528c-be2f-4d6e-8843-02458d09e26d","Type":"ContainerDied","Data":"d82319dce68ca5a640bd2529e3fcca49c137e59b71819b494a2dfbfa2e4a4f7a"} Oct 31 00:40:04.997521 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.997504 4471 scope.go:122] "RemoveContainer" containerID="fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a" Oct 31 00:40:04.997857 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:04.997833 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a\": container with ID starting with fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a not found: ID does not exist" containerID="fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a" Oct 31 00:40:04.997952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.997868 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a"} err="failed to get container status \"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a\": rpc error: code = NotFound desc = could not find container \"fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a\": container with ID starting with fb444ab37746f46fb045c35ecbe65cbdc56a71263797dcc2c5ab27bf81bb907a not found: ID does not exist" Oct 31 00:40:04.997952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:04.997893 4471 scope.go:122] "RemoveContainer" containerID="9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72" Oct 31 00:40:05.007394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.007373 4471 scope.go:122] "RemoveContainer" containerID="9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72" Oct 31 00:40:05.007697 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:05.007679 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72\": container with ID starting with 9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72 not found: ID does not exist" containerID="9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72" Oct 31 00:40:05.007746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.007705 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72"} err="failed to get container status \"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72\": rpc error: code = NotFound desc = could not find container \"9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72\": container with ID starting with 9ecaa91fba661a2d8cf94d26528fef0607a7b797686a433b2ea41268c6f57d72 not found: ID does not exist" Oct 31 00:40:05.152615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.152579 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:40:05.156420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.156399 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7b593fa-5c65dc776c-m8ptm"] Oct 31 00:40:05.209190 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.209158 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209a528c-be2f-4d6e-8843-02458d09e26d" path="/var/lib/kubelet/pods/209a528c-be2f-4d6e-8843-02458d09e26d/volumes" Oct 31 00:40:05.287001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.286970 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:40:05.317032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:05.317002 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e7b593fa-6cffcfbb4c-fr9cc"] Oct 31 00:40:07.209957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:07.209923 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672541b4-3959-4f5f-bf63-faf9b3753a66" path="/var/lib/kubelet/pods/672541b4-3959-4f5f-bf63-faf9b3753a66/volumes" Oct 31 00:40:07.882245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:07.882203 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:08.811009 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:08.810976 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:08.961181 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:08.961137 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:12.975910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:12.975828 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:17.979446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:17.979405 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:18.921994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:18.921965 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:19.099709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:19.099670 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:22.280304 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.280274 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:40:22.280737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.280478 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerName="maas-api" containerID="cri-o://92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e" gracePeriod=30 Oct 31 00:40:22.311907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.311871 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.67:8443/health\": dial tcp 10.133.0.67:8443: connect: connection refused" Oct 31 00:40:22.481052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.480946 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:22.653217 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.653183 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:40:22.712551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.712162 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\") pod \"38c05a9a-f25e-4ea6-8553-f995fce69e67\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " Oct 31 00:40:22.712551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.712304 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") pod \"38c05a9a-f25e-4ea6-8553-f995fce69e67\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " Oct 31 00:40:22.712551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.712335 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\") pod \"38c05a9a-f25e-4ea6-8553-f995fce69e67\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " Oct 31 00:40:22.712551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.712509 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\" (UniqueName: \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\") pod \"38c05a9a-f25e-4ea6-8553-f995fce69e67\" (UID: \"38c05a9a-f25e-4ea6-8553-f995fce69e67\") " Oct 31 00:40:22.714042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.713998 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca" pod "38c05a9a-f25e-4ea6-8553-f995fce69e67" (UID: "38c05a9a-f25e-4ea6-8553-f995fce69e67"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:22.716815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.716784 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs" pod "38c05a9a-f25e-4ea6-8553-f995fce69e67" (UID: "38c05a9a-f25e-4ea6-8553-f995fce69e67"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:22.722041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.721996 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf" pod "38c05a9a-f25e-4ea6-8553-f995fce69e67" (UID: "38c05a9a-f25e-4ea6-8553-f995fce69e67"). InnerVolumeSpecName "kube-api-access-vvmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:22.723927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.723902 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls" pod "38c05a9a-f25e-4ea6-8553-f995fce69e67" (UID: "38c05a9a-f25e-4ea6-8553-f995fce69e67"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:22.815629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.815361 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.815629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.815400 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38c05a9a-f25e-4ea6-8553-f995fce69e67-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.815629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.815415 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38c05a9a-f25e-4ea6-8553-f995fce69e67-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:22.815629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:22.815429 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmkf\" (UniqueName: \"kubernetes.io/projected/38c05a9a-f25e-4ea6-8553-f995fce69e67-kube-api-access-vvmkf\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:23.116441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.116409 4471 generic.go:356] "Generic (PLEG): container finished" podID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerID="92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e" exitCode=0 Oct 31 00:40:23.116698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.116465 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" Oct 31 00:40:23.116698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.116503 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" event={"ID":"38c05a9a-f25e-4ea6-8553-f995fce69e67","Type":"ContainerDied","Data":"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e"} Oct 31 00:40:23.116698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.116601 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws" event={"ID":"38c05a9a-f25e-4ea6-8553-f995fce69e67","Type":"ContainerDied","Data":"857746feb637a3cf219d7d9b07db8814dc5470102083ef90492dcc7e51fedacd"} Oct 31 00:40:23.116698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.116628 4471 scope.go:122] "RemoveContainer" containerID="92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e" Oct 31 00:40:23.127756 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.127735 4471 scope.go:122] "RemoveContainer" containerID="92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e" Oct 31 00:40:23.128046 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:23.128027 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e\": container with ID starting with 92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e not found: ID does not exist" containerID="92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e" Oct 31 00:40:23.128098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.128055 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e"} err="failed to get container status \"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e\": rpc error: code = NotFound desc = could not find container \"92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e\": container with ID starting with 92f6ae7c42087a772d2dccb12ecec7c1fee968ff35f612a553127db27072da4e not found: ID does not exist" Oct 31 00:40:23.282416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.282376 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:40:23.285004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.284983 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26fb7237-6bfd86bbb9-xcgws"] Oct 31 00:40:23.298683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.298659 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:23.964074 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:23.964038 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:24.193412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:24.193369 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:24.350362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:24.350269 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:24.500151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:24.500114 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:24.736447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:24.736415 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:25.209641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:25.209609 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" path="/var/lib/kubelet/pods/38c05a9a-f25e-4ea6-8553-f995fce69e67/volumes" Oct 31 00:40:25.449914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:25.449876 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:26.165006 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:26.164844 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:26.459700 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:26.459596 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:28.792983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:28.792951 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:40:28.793391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:28.793160 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" podUID="b8e07179-f097-43c7-a25e-e9a766dde88f" containerName="maas-api" containerID="cri-o://79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d" gracePeriod=30 Oct 31 00:40:28.896375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:28.896327 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:29.102455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.102427 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:40:29.183678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.183640 4471 generic.go:356] "Generic (PLEG): container finished" podID="b8e07179-f097-43c7-a25e-e9a766dde88f" containerID="79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d" exitCode=0 Oct 31 00:40:29.183678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.183681 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" Oct 31 00:40:29.183952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.183725 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" event={"ID":"b8e07179-f097-43c7-a25e-e9a766dde88f","Type":"ContainerDied","Data":"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d"} Oct 31 00:40:29.183952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.183785 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8" event={"ID":"b8e07179-f097-43c7-a25e-e9a766dde88f","Type":"ContainerDied","Data":"5c2000f62ec4553d19ccde32fca911a91a12d879dd88e5e308a54ecefba8b769"} Oct 31 00:40:29.183952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.183821 4471 scope.go:122] "RemoveContainer" containerID="79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d" Oct 31 00:40:29.195408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.195385 4471 scope.go:122] "RemoveContainer" containerID="79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d" Oct 31 00:40:29.195812 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:29.195791 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d\": container with ID starting with 79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d not found: ID does not exist" containerID="79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d" Oct 31 00:40:29.195872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.195822 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d"} err="failed to get container status \"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d\": rpc error: code = NotFound desc = could not find container \"79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d\": container with ID starting with 79a6768a78916b257e6b9594668b8ed086876eaef4cb9f7c63f50089dc60571d not found: ID does not exist" Oct 31 00:40:29.224857 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.224822 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") pod \"b8e07179-f097-43c7-a25e-e9a766dde88f\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " Oct 31 00:40:29.225048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.224991 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\") pod \"b8e07179-f097-43c7-a25e-e9a766dde88f\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " Oct 31 00:40:29.225048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.225026 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\" (UniqueName: \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\") pod \"b8e07179-f097-43c7-a25e-e9a766dde88f\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " Oct 31 00:40:29.225172 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.225109 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\") pod \"b8e07179-f097-43c7-a25e-e9a766dde88f\" (UID: \"b8e07179-f097-43c7-a25e-e9a766dde88f\") " Oct 31 00:40:29.225725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.225688 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca" pod "b8e07179-f097-43c7-a25e-e9a766dde88f" (UID: "b8e07179-f097-43c7-a25e-e9a766dde88f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:29.225851 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.225805 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8e07179-f097-43c7-a25e-e9a766dde88f-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:29.230040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.228002 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls" pod "b8e07179-f097-43c7-a25e-e9a766dde88f" (UID: "b8e07179-f097-43c7-a25e-e9a766dde88f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:29.230040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.228159 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs" pod "b8e07179-f097-43c7-a25e-e9a766dde88f" (UID: "b8e07179-f097-43c7-a25e-e9a766dde88f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:29.230040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.228165 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp" pod "b8e07179-f097-43c7-a25e-e9a766dde88f" (UID: "b8e07179-f097-43c7-a25e-e9a766dde88f"). InnerVolumeSpecName "kube-api-access-rhzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:29.327718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.327503 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:29.327718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.327569 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8e07179-f097-43c7-a25e-e9a766dde88f-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:29.327718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.327588 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzvp\" (UniqueName: \"kubernetes.io/projected/b8e07179-f097-43c7-a25e-e9a766dde88f-kube-api-access-rhzvp\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:29.537076 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.537038 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:29.608503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.608415 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:40:29.613515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.613493 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7b593fa-6dd5bff657-w64k8"] Oct 31 00:40:29.748824 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.748771 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:30.000674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:29.997832 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:30.845769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:30.845733 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:31.064908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:31.064869 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:31.212281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:31.212237 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e07179-f097-43c7-a25e-e9a766dde88f" path="/var/lib/kubelet/pods/b8e07179-f097-43c7-a25e-e9a766dde88f/volumes" Oct 31 00:40:31.224980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:31.224915 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:32.042948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:32.042906 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:32.745037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:32.745003 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.362211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.362174 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:40:33.363022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.362997 4471 cpu_manager.go:414] "containerMap: removing container" podUID="209a528c-be2f-4d6e-8843-02458d09e26d" containerName="payload-pre-processing" Oct 31 00:40:33.363212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363198 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209a528c-be2f-4d6e-8843-02458d09e26d" containerName="payload-pre-processing" Oct 31 00:40:33.363339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363319 4471 cpu_manager.go:414] "containerMap: removing container" podUID="b8e07179-f097-43c7-a25e-e9a766dde88f" containerName="maas-api" Oct 31 00:40:33.363339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363334 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e07179-f097-43c7-a25e-e9a766dde88f" containerName="maas-api" Oct 31 00:40:33.363503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363364 4471 cpu_manager.go:414] "containerMap: removing container" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerName="maas-api" Oct 31 00:40:33.363503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363371 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerName="maas-api" Oct 31 00:40:33.363503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363393 4471 cpu_manager.go:414] "containerMap: removing container" podUID="672541b4-3959-4f5f-bf63-faf9b3753a66" containerName="payload-processing" Oct 31 00:40:33.363503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363399 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672541b4-3959-4f5f-bf63-faf9b3753a66" containerName="payload-processing" Oct 31 00:40:33.363746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363566 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c05a9a-f25e-4ea6-8553-f995fce69e67" containerName="maas-api" Oct 31 00:40:33.363746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363581 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="672541b4-3959-4f5f-bf63-faf9b3753a66" containerName="payload-processing" Oct 31 00:40:33.363746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363593 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="209a528c-be2f-4d6e-8843-02458d09e26d" containerName="payload-pre-processing" Oct 31 00:40:33.363746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.363608 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e07179-f097-43c7-a25e-e9a766dde88f" containerName="maas-api" Oct 31 00:40:33.369245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.369220 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.424146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.424114 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.431976 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.431943 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:40:33.474016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.473986 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-74c6ba99-openshift-default-dockercfg-mnr9m\"" Oct 31 00:40:33.493254 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493225 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493254 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493258 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493492 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493285 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493492 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493460 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493598 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw8s\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493654 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493673 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493716 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.493829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.493813 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.624730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.620622 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.642992 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643150 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643175 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643221 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643297 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643388 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw8s\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643462 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643486 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.643516 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.647555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.645866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.673554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.664886 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.673554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.665128 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.673554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.666051 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.673554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.669856 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.692125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.687082 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.693106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.693078 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.727435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.727400 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw8s\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.732207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.732173 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\") pod \"e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:33.863014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:33.862970 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:34.007555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.004178 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:34.049473 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.046458 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:34.285607 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:40:34.284770 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:40:34.556852 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:40:34.556818 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277be621_e3ed_4388_b43f_6bde98ebcf50.slice/crio-173589bd3ac5800cd47bdf07200f40a5521a2ff492312d3679144958bf251fab WatchSource:0}: Error finding container 173589bd3ac5800cd47bdf07200f40a5521a2ff492312d3679144958bf251fab: Status 404 returned error can't find the container with id 173589bd3ac5800cd47bdf07200f40a5521a2ff492312d3679144958bf251fab Oct 31 00:40:34.559519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.559495 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:40:34.559870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.559842 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:34.559920 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.559906 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:34.559957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:34.559931 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:35.232599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:35.232560 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerStarted","Data":"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5"} Oct 31 00:40:35.233127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:35.232619 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerStarted","Data":"173589bd3ac5800cd47bdf07200f40a5521a2ff492312d3679144958bf251fab"} Oct 31 00:40:35.471812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:35.471760 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podStartSLOduration=2.471744644 podStartE2EDuration="2.471744644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:35.420829161 +0000 UTC m=+1562.346098113" watchObservedRunningTime="2026-10-31 00:40:35.471744644 +0000 UTC m=+1562.397013650" Oct 31 00:40:35.915022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:35.914980 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:40:35.915274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:35.915247 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" podUID="d296dc07-39f1-46ac-9433-734ad78ffcfe" containerName="maas-api" containerID="cri-o://425e77054cc43bb63e7ede8e941fbb244f2604a8bcd5b93125da948346402257" gracePeriod=30 Oct 31 00:40:36.005677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.005641 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:36.011567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.011523 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:36.048564 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.048508 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:36.343594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.337801 4471 generic.go:356] "Generic (PLEG): container finished" podID="d296dc07-39f1-46ac-9433-734ad78ffcfe" containerID="425e77054cc43bb63e7ede8e941fbb244f2604a8bcd5b93125da948346402257" exitCode=0 Oct 31 00:40:36.345364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.344671 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" event={"ID":"d296dc07-39f1-46ac-9433-734ad78ffcfe","Type":"ContainerDied","Data":"425e77054cc43bb63e7ede8e941fbb244f2604a8bcd5b93125da948346402257"} Oct 31 00:40:36.345364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.344746 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:36.356706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.353268 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:40:36.366565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.363321 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:40:36.412346 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.412318 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\") pod \"d296dc07-39f1-46ac-9433-734ad78ffcfe\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " Oct 31 00:40:36.412581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.412412 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") pod \"d296dc07-39f1-46ac-9433-734ad78ffcfe\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " Oct 31 00:40:36.412581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.412450 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\") pod \"d296dc07-39f1-46ac-9433-734ad78ffcfe\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " Oct 31 00:40:36.412704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.412597 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\" (UniqueName: \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\") pod \"d296dc07-39f1-46ac-9433-734ad78ffcfe\" (UID: \"d296dc07-39f1-46ac-9433-734ad78ffcfe\") " Oct 31 00:40:36.414090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.414056 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca" pod "d296dc07-39f1-46ac-9433-734ad78ffcfe" (UID: "d296dc07-39f1-46ac-9433-734ad78ffcfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:36.416183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.416159 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs" pod "d296dc07-39f1-46ac-9433-734ad78ffcfe" (UID: "d296dc07-39f1-46ac-9433-734ad78ffcfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:36.416851 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.416829 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls" pod "d296dc07-39f1-46ac-9433-734ad78ffcfe" (UID: "d296dc07-39f1-46ac-9433-734ad78ffcfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:36.418184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.418153 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2" pod "d296dc07-39f1-46ac-9433-734ad78ffcfe" (UID: "d296dc07-39f1-46ac-9433-734ad78ffcfe"). InnerVolumeSpecName "kube-api-access-948t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:36.519608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.516385 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d296dc07-39f1-46ac-9433-734ad78ffcfe-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:36.519608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.516421 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:36.519608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.516436 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d296dc07-39f1-46ac-9433-734ad78ffcfe-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:36.519608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:36.516450 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948t2\" (UniqueName: \"kubernetes.io/projected/d296dc07-39f1-46ac-9433-734ad78ffcfe-kube-api-access-948t2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:37.346027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.345995 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" Oct 31 00:40:37.346027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.346001 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65" event={"ID":"d296dc07-39f1-46ac-9433-734ad78ffcfe","Type":"ContainerDied","Data":"4dce241d8868b1a508fa7fbcc9df03ee33da93c6b120097064e73538235af828"} Oct 31 00:40:37.346603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.346087 4471 scope.go:122] "RemoveContainer" containerID="425e77054cc43bb63e7ede8e941fbb244f2604a8bcd5b93125da948346402257" Oct 31 00:40:37.520079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.519961 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:40:37.570483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.570400 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49bb6b1a-786c6bb464-2ql65"] Oct 31 00:40:37.582592 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.578262 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:37.783273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.782725 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:37.983914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:37.983872 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:38.247599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:38.247502 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:38.410554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:38.406703 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:38.728222 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:38.728189 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:39.074501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:39.074413 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:39.229763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:39.229726 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d296dc07-39f1-46ac-9433-734ad78ffcfe" path="/var/lib/kubelet/pods/d296dc07-39f1-46ac-9433-734ad78ffcfe/volumes" Oct 31 00:40:40.152999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:40.152955 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:40.315541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:40.315493 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:41.078241 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.078203 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:41.254863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.254823 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:41.455690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.455656 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:40:41.456652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.456616 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d296dc07-39f1-46ac-9433-734ad78ffcfe" containerName="maas-api" Oct 31 00:40:41.456652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.456653 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d296dc07-39f1-46ac-9433-734ad78ffcfe" containerName="maas-api" Oct 31 00:40:41.456819 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.456807 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d296dc07-39f1-46ac-9433-734ad78ffcfe" containerName="maas-api" Oct 31 00:40:41.462178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.462150 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.517359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.517327 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:40:41.540815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.540778 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.540991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.540848 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5sd9\" (UniqueName: \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.562834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.562806 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-74c6ba99\"" Oct 31 00:40:41.563013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.562855 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-74c6ba99-dockercfg-vknpv\"" Oct 31 00:40:41.625208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.625160 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:40:41.629787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.629755 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.658743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.649170 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.660166 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.652646 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.664707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.660390 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5sd9\" (UniqueName: \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.693219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.693173 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:40:41.722183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.722100 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5sd9\" (UniqueName: \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\") pod \"payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.766419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.766383 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzgw\" (UniqueName: \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.766608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.766543 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.777258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.777226 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:41.867791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.867755 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzgw\" (UniqueName: \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.867991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.867888 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.868606 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.868580 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.932977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.932945 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzgw\" (UniqueName: \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\") pod \"payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:41.967042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:41.967009 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:42.161325 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:40:42.161283 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dac9294_8c4a_42fd_af87_fc4afc0a2f38.slice/crio-053460e84f91d7677fb6b561bca1d3041b12db2376c3a03298361c031381e8a1 WatchSource:0}: Error finding container 053460e84f91d7677fb6b561bca1d3041b12db2376c3a03298361c031381e8a1: Status 404 returned error can't find the container with id 053460e84f91d7677fb6b561bca1d3041b12db2376c3a03298361c031381e8a1 Oct 31 00:40:42.161815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.161747 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:40:42.340007 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:40:42.339977 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8935dc_7188_4a77_af06_016507fb84b9.slice/crio-0b341cacef31a9488c6802b997902e0cd3708c1138174ade6a8f44e778893abd WatchSource:0}: Error finding container 0b341cacef31a9488c6802b997902e0cd3708c1138174ade6a8f44e778893abd: Status 404 returned error can't find the container with id 0b341cacef31a9488c6802b997902e0cd3708c1138174ade6a8f44e778893abd Oct 31 00:40:42.352599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.352566 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:40:42.391439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.391394 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" event={"ID":"5dac9294-8c4a-42fd-af87-fc4afc0a2f38","Type":"ContainerStarted","Data":"1f390b002b6da8f6f4ca55ac0cb8f7b5334b4db0ff819042f20b992739891223"} Oct 31 00:40:42.391657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.391473 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:42.391657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.391496 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" event={"ID":"5dac9294-8c4a-42fd-af87-fc4afc0a2f38","Type":"ContainerStarted","Data":"053460e84f91d7677fb6b561bca1d3041b12db2376c3a03298361c031381e8a1"} Oct 31 00:40:42.392809 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.392781 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" event={"ID":"ae8935dc-7188-4a77-af06-016507fb84b9","Type":"ContainerStarted","Data":"0b341cacef31a9488c6802b997902e0cd3708c1138174ade6a8f44e778893abd"} Oct 31 00:40:42.514610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:42.514493 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" podStartSLOduration=1.514480015 podStartE2EDuration="1.514480015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:42.512141358 +0000 UTC m=+1569.437410330" watchObservedRunningTime="2026-10-31 00:40:42.514480015 +0000 UTC m=+1569.439748979" Oct 31 00:40:43.400912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:43.400820 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" event={"ID":"ae8935dc-7188-4a77-af06-016507fb84b9","Type":"ContainerStarted","Data":"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e"} Oct 31 00:40:43.401330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:43.401071 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:43.518307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:43.518248 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" podStartSLOduration=2.518232704 podStartE2EDuration="2.518232704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:43.516512406 +0000 UTC m=+1570.441781358" watchObservedRunningTime="2026-10-31 00:40:43.518232704 +0000 UTC m=+1570.443501657" Oct 31 00:40:43.777835 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:43.777746 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:48.941578 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:48.930491 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:53.403960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.403927 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:40:53.852970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.852881 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:53.859355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.859332 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr"] Oct 31 00:40:53.863285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.863268 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.941948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.941908 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.941977 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ca558-2510-451e-9fb4-8c261ed2933d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942101 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942121 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942143 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dfp\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-kube-api-access-95dfp\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942212 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ca558-2510-451e-9fb4-8c261ed2933d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942310 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942355 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.942441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.942410 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:53.973061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:53.973021 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr"] Oct 31 00:40:54.043882 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.043844 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.043922 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ca558-2510-451e-9fb4-8c261ed2933d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.043977 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044002 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044031 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dfp\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-kube-api-access-95dfp\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044075 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ca558-2510-451e-9fb4-8c261ed2933d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044164 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044226 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044263 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044331 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044358 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044518 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044783 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.044913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.044840 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/668ca558-2510-451e-9fb4-8c261ed2933d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.046601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.046573 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/668ca558-2510-451e-9fb4-8c261ed2933d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.046891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.046869 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/668ca558-2510-451e-9fb4-8c261ed2933d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.107303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.107218 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.107501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.107478 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dfp\" (UniqueName: \"kubernetes.io/projected/668ca558-2510-451e-9fb4-8c261ed2933d-kube-api-access-95dfp\") pod \"maas-default-gateway-openshift-default-6886b96f9-cdzdr\" (UID: \"668ca558-2510-451e-9fb4-8c261ed2933d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.180717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.180675 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:54.411502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.411473 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:40:54.604837 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:40:54.604807 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668ca558_2510_451e_9fb4_8c261ed2933d.slice/crio-3eed396f35a717e5a294f6a5d6dc1d1e854ff98cc1d6953fcc84d8bd4eb60dc8 WatchSource:0}: Error finding container 3eed396f35a717e5a294f6a5d6dc1d1e854ff98cc1d6953fcc84d8bd4eb60dc8: Status 404 returned error can't find the container with id 3eed396f35a717e5a294f6a5d6dc1d1e854ff98cc1d6953fcc84d8bd4eb60dc8 Oct 31 00:40:54.606665 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.606641 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr"] Oct 31 00:40:54.607671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.607640 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:54.607746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.607702 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:54.607746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:54.607728 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:40:55.488403 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:55.488366 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" event={"ID":"668ca558-2510-451e-9fb4-8c261ed2933d","Type":"ContainerStarted","Data":"020cfc321eb52f77d98c6d24cd23577b0940238761bd58dc850759dddc20ffaa"} Oct 31 00:40:55.488863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:55.488421 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" event={"ID":"668ca558-2510-451e-9fb4-8c261ed2933d","Type":"ContainerStarted","Data":"3eed396f35a717e5a294f6a5d6dc1d1e854ff98cc1d6953fcc84d8bd4eb60dc8"} Oct 31 00:40:55.711957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:55.711904 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podStartSLOduration=2.711891296 podStartE2EDuration="2.711891296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:55.662070997 +0000 UTC m=+1582.587339952" watchObservedRunningTime="2026-10-31 00:40:55.711891296 +0000 UTC m=+1582.637160245" Oct 31 00:40:56.181834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:56.181792 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:40:56.183979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:56.183947 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:40:56.184117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:56.184000 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:40:57.006081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:57.006047 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:57.182083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:57.182039 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:40:57.182290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:57.182125 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:40:58.183258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:58.183220 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:40:58.183916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:58.183884 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:40:58.293224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:58.293189 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:59.182078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:59.182027 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:40:59.182265 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:59.182098 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:40:59.236554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:40:59.236493 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:00.181949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:00.181914 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:00.182164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:00.181970 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:41:01.181598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:01.181560 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:01.182014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:01.181628 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:41:01.327253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:01.327183 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:02.181445 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:02.181402 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:41:02.181690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:02.181469 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 00:41:02.615805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:02.615681 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:03.243419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:03.243371 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:41:03.243950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:03.243847 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:41:03.244747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:03.244728 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" Oct 31 00:41:03.458297 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:03.458258 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:04.281453 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:41:04.281415 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:41:04.401919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:04.401882 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:04.870293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:04.870260 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.339094 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:09.339049 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:09.497619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:09.497583 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:14.351141 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:14.351106 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:14.522455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:14.522421 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:14.709239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:14.709202 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:19.696998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:19.696963 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:21.694514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:21.694466 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:22.342832 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.342796 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 00:41:22.346821 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.346803 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.403945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.403910 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-4b86e4-openshift-default-dockercfg-5q5j8\"" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456308 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456360 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njms5\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456406 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456496 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456521 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456681 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456769 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.457552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.456792 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.464912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.464187 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 00:41:22.504589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.490980 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:22.575795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.558470 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576200 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576247 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576396 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576422 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njms5\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576479 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576611 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576685 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.576752 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.576712 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.577918 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.577891 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.578294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.578156 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.581124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.578761 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.581124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.579589 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.582016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.581551 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.602145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.601659 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.602145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.601980 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.640834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.640799 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.646921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.646863 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njms5\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\") pod \"e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.681624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.681587 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:22.914769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:22.914725 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.117836 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:41:23.117795 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026a7b63_8e3d_4749_b2e4_a74a5c1c4f75.slice/crio-75cafcb74067e211d9f30df96b251116da3bb5f11aff715ebf2ae3c232b84fa5 WatchSource:0}: Error finding container 75cafcb74067e211d9f30df96b251116da3bb5f11aff715ebf2ae3c232b84fa5: Status 404 returned error can't find the container with id 75cafcb74067e211d9f30df96b251116da3bb5f11aff715ebf2ae3c232b84fa5 Oct 31 00:41:23.120119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.120094 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 00:41:23.120283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.120251 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:23.120353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.120317 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:23.120353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.120346 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:23.438364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.438274 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.644881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.644846 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:23.676745 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.676708 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerStarted","Data":"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258"} Oct 31 00:41:23.676925 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.676764 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerStarted","Data":"75cafcb74067e211d9f30df96b251116da3bb5f11aff715ebf2ae3c232b84fa5"} Oct 31 00:41:23.682065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.682045 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:23.801618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:23.801473 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podStartSLOduration=1.8014528909999998 podStartE2EDuration="1.801452891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:23.798128975 +0000 UTC m=+1610.723397928" watchObservedRunningTime="2026-10-31 00:41:23.801452891 +0000 UTC m=+1610.726721844" Oct 31 00:41:24.255950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.255914 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:41:24.264558 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.262129 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303132 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303291 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303324 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303357 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4cs\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303401 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303453 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303549 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303579 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.312552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.303607 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.455387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.455127 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:24.457082 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.457044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.457431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.457294 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.457757 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.457637 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.458013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.457918 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.458413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.458299 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.458503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.458484 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.458720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.458629 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4cs\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.459001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.458883 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.459161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.459144 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.512740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.498212 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.512740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.500190 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.512740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.511002 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f0651cac-openshift-default-dockercfg-msrcp\"" Oct 31 00:41:24.517811 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.517786 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.530554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.530291 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.530734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.530587 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:41:24.543556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.530907 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.543556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.531503 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.543556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.532474 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.593335 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.593301 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.593582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.593519 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4cs\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\") pod \"e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.620658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.620622 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:24.689525 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.689497 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:24.943333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:24.943294 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:25.116012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.115707 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:41:25.119354 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:41:25.119151 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6de6233_4820_4d44_84a5_b6d90b273a46.slice/crio-7072bc1b9c91954c084c4a0f0046ff48c4ef4326da0545acf03f9ab111ace0a8 WatchSource:0}: Error finding container 7072bc1b9c91954c084c4a0f0046ff48c4ef4326da0545acf03f9ab111ace0a8: Status 404 returned error can't find the container with id 7072bc1b9c91954c084c4a0f0046ff48c4ef4326da0545acf03f9ab111ace0a8 Oct 31 00:41:25.122983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.122943 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:25.123124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.123063 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:25.123124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.123110 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:41:25.187587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.180722 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:25.377377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.377331 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:25.691482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.691438 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerStarted","Data":"8c178d33aa86580391153f1a4bbf0d7e25806b61578011d399e75b3afc4473c6"} Oct 31 00:41:25.691730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.691509 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerStarted","Data":"7072bc1b9c91954c084c4a0f0046ff48c4ef4326da0545acf03f9ab111ace0a8"} Oct 31 00:41:25.691788 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.691743 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:25.692990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.692968 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:41:25.819042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:25.818985 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podStartSLOduration=1.818969609 podStartE2EDuration="1.818969609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:25.816586716 +0000 UTC m=+1612.741855670" watchObservedRunningTime="2026-10-31 00:41:25.818969609 +0000 UTC m=+1612.744238561" Oct 31 00:41:26.621592 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:26.621559 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:26.627117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:26.627093 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:26.698034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:26.698001 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:26.698997 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:26.698979 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:41:26.843553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:26.843489 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.109761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:27.109726 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.400522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:27.400489 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:29.605614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:29.605578 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:41:29.606037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:29.605804 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" containerID="cri-o://7e91cfb86f4f5922225861432be59f0bfd452c43399dbe9f33810c429f2bf3a8" gracePeriod=30 Oct 31 00:41:29.754504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:29.754465 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:29.952789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:29.952755 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.140707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:30.140671 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.358879 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:30.358781 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.572644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:30.572600 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.768029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:30.767998 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:33.024078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:33.024027 4471 patch_prober.go:29] interesting pod/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:41:33.024486 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:33.024132 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:41:34.283439 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:41:34.283405 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:41:34.756895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.756858 4471 generic.go:356] "Generic (PLEG): container finished" podID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerID="7e91cfb86f4f5922225861432be59f0bfd452c43399dbe9f33810c429f2bf3a8" exitCode=0 Oct 31 00:41:34.757068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.756935 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" event={"ID":"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8","Type":"ContainerDied","Data":"7e91cfb86f4f5922225861432be59f0bfd452c43399dbe9f33810c429f2bf3a8"} Oct 31 00:41:34.894763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.894727 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:41:34.952480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952445 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952500 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952602 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952709 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952763 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952850 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:34.952945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952869 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:34.952945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952872 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.952945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.952912 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:34.953160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.953025 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.953160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.953085 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\") pod \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\" (UID: \"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8\") " Oct 31 00:41:34.953463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.953435 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:34.953682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.953659 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:34.953965 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.953937 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:34.954133 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.954108 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:34.954227 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.954135 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:34.954227 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.954150 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:34.954227 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.954164 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:34.955228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.955204 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "kube-api-access-j9dww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:34.955336 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.955241 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:34.955336 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.955242 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:41:34.955336 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:34.955264 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy" pod "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" (UID: "03880da0-5a1c-4d49-8a4f-eb28c35dd9e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:35.055895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.055856 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dww\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-kube-api-access-j9dww\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:35.055895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.055894 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:35.056111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.055910 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:35.056111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.055923 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:35.764144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.764110 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" event={"ID":"03880da0-5a1c-4d49-8a4f-eb28c35dd9e8","Type":"ContainerDied","Data":"982ee0263a90b4489286a2a00b7fe540909b732043f9627f5446d175f678d1b4"} Oct 31 00:41:35.764144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.764145 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69" Oct 31 00:41:35.764722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.764180 4471 scope.go:122] "RemoveContainer" containerID="7e91cfb86f4f5922225861432be59f0bfd452c43399dbe9f33810c429f2bf3a8" Oct 31 00:41:35.984448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:35.984411 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:41:36.035937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:36.035854 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-9vg69"] Oct 31 00:41:36.180183 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:36.180139 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:37.211688 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.211656 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" path="/var/lib/kubelet/pods/03880da0-5a1c-4d49-8a4f-eb28c35dd9e8/volumes" Oct 31 00:41:37.256993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.256957 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:37.438274 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.438171 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:37.683583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.683548 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:41:37.684449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.684430 4471 cpu_manager.go:414] "containerMap: removing container" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" Oct 31 00:41:37.684517 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.684454 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" Oct 31 00:41:37.684666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.684653 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="03880da0-5a1c-4d49-8a4f-eb28c35dd9e8" containerName="istio-proxy" Oct 31 00:41:37.690121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.690096 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:37.744982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.744951 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:41:37.790678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.790643 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f0651cac\"" Oct 31 00:41:37.814224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.814189 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:41:37.818213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.818193 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:37.842073 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.842040 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:37.842230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.842102 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz98d\" (UniqueName: \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:37.875114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.875084 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f0651cac-dockercfg-l7w5s\"" Oct 31 00:41:37.910557 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.889277 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:41:37.943984 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.943888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxkp\" (UniqueName: \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:37.944167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.944034 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:37.944238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.944174 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:37.944238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.944224 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz98d\" (UniqueName: \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:37.945288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:37.945258 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:38.030553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.030471 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz98d\" (UniqueName: \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\") pod \"payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:38.045441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.045403 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.045981 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.045953 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxkp\" (UniqueName: \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.047422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.047400 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.113552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.111039 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxkp\" (UniqueName: \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\") pod \"payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.131928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.131850 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.306744 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.306458 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:38.621106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.621067 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:41:38.769404 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:41:38.769107 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc528a86_c6b5_4664_a43c_f1016fa1aff9.slice/crio-7307b21ac60eaa54b0af5cc46d6fe2c105920031077ac4211c6662f6a406ab24 WatchSource:0}: Error finding container 7307b21ac60eaa54b0af5cc46d6fe2c105920031077ac4211c6662f6a406ab24: Status 404 returned error can't find the container with id 7307b21ac60eaa54b0af5cc46d6fe2c105920031077ac4211c6662f6a406ab24 Oct 31 00:41:38.772238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.772209 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:41:38.795350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.794564 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" event={"ID":"24b5dec0-e590-4809-b749-e25802054e92","Type":"ContainerStarted","Data":"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4"} Oct 31 00:41:38.795350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.794634 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" event={"ID":"24b5dec0-e590-4809-b749-e25802054e92","Type":"ContainerStarted","Data":"abf31ecdce4a9e167ad1de589912f4f39567242366c7ff27e4a590e3e4c16efc"} Oct 31 00:41:38.795350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.794848 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:38.796117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.796085 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" event={"ID":"dc528a86-c6b5-4664-a43c-f1016fa1aff9","Type":"ContainerStarted","Data":"7307b21ac60eaa54b0af5cc46d6fe2c105920031077ac4211c6662f6a406ab24"} Oct 31 00:41:38.920201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:38.920141 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" podStartSLOduration=1.9201276109999998 podStartE2EDuration="1.920127611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:38.917498737 +0000 UTC m=+1625.842767690" watchObservedRunningTime="2026-10-31 00:41:38.920127611 +0000 UTC m=+1625.845396562" Oct 31 00:41:39.837404 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:39.836667 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" event={"ID":"dc528a86-c6b5-4664-a43c-f1016fa1aff9","Type":"ContainerStarted","Data":"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2"} Oct 31 00:41:39.837404 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:39.836794 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:41.537081 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:41.537021 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:46.649319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:46.649271 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:49.840982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:49.840950 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:41:49.959504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:49.959452 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" podStartSLOduration=12.959438524 podStartE2EDuration="12.959438524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:39.981892745 +0000 UTC m=+1626.907161710" watchObservedRunningTime="2026-10-31 00:41:49.959438524 +0000 UTC m=+1636.884707477" Oct 31 00:41:50.847346 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:50.847306 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:41:51.678073 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:51.678034 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:56.728646 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:41:56.728607 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:01.787004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:01.786961 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:04.281584 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:42:04.281523 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:42:06.781358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:06.781322 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:11.885485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:11.885453 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.184430 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:17.184342 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.342488 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:17.342453 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:17.516962 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:17.516878 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:22.637387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:22.637352 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:25.182356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:25.182317 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:25.400891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:25.400854 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:27.552131 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:27.552093 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:27.954125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:27.954092 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.182049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:28.182013 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.358524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:28.358429 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.579323 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:28.579289 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:28.782176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:28.782139 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:29.723283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:29.723249 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:30.057036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:30.056942 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq"] Oct 31 00:42:30.099587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:30.099551 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdda52da-p79sq"] Oct 31 00:42:31.210035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:31.209999 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704a490e-9a23-4aff-b3df-4d1afc7a79ae" path="/var/lib/kubelet/pods/704a490e-9a23-4aff-b3df-4d1afc7a79ae/volumes" Oct 31 00:42:32.645147 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:32.645108 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:32.830588 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:32.830518 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.092876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:33.092798 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.503959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:33.503922 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:33.981766 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:33.981730 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:34.137480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.137431 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:42:34.147410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.147377 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.232609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.224982 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:42:34.232609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.226617 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247119 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflpw\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247196 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247229 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247309 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247374 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247427 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247497 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247539 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.249552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.247598 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.274616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.273735 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d5f4de7-openshift-default-dockercfg-q5jrh\"" Oct 31 00:42:34.318663 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.315826 4471 generic.go:356] "Generic (PLEG): container finished" podID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerID="76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5" exitCode=0 Oct 31 00:42:34.320891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.318979 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerDied","Data":"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5"} Oct 31 00:42:34.320891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.319544 4471 scope.go:122] "RemoveContainer" containerID="76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5" Oct 31 00:42:34.323774 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.323732 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:34.324042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.324010 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:34.324234 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.324195 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:34.353052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.351232 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.353052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.351413 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflpw\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.353052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.351487 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.353052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.352264 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.353052 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.352997 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.353923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.351521 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354136 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354201 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354259 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354341 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354369 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.354761 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.355417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.355014 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.356448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.356395 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.359915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.359858 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.368739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.367845 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.413084 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:42:34.410196 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:42:34.415075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.414601 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflpw\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.425155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.425119 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.532408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.531364 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:34.733261 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.733222 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:34.991091 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:42:34.991055 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47373da_96ae_46fc_be7e_ae1f5f014d7e.slice/crio-73fa4ebf19ec67331f1414e9d09728e94bf6e6be2cab8f455923dde180b88687 WatchSource:0}: Error finding container 73fa4ebf19ec67331f1414e9d09728e94bf6e6be2cab8f455923dde180b88687: Status 404 returned error can't find the container with id 73fa4ebf19ec67331f1414e9d09728e94bf6e6be2cab8f455923dde180b88687 Oct 31 00:42:34.992866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.992763 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:42:34.993913 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.993875 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:34.994025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.993956 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:34.994025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:34.994001 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:35.329121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:35.329019 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerStarted","Data":"f1b58f2b6105866ec3bd2c124610f61131997d742f96f53ef033b8e3eb80cedf"} Oct 31 00:42:35.329121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:35.329091 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerStarted","Data":"73fa4ebf19ec67331f1414e9d09728e94bf6e6be2cab8f455923dde180b88687"} Oct 31 00:42:35.331740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:35.331712 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerStarted","Data":"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f"} Oct 31 00:42:35.451338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:35.451273 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podStartSLOduration=1.451257152 podStartE2EDuration="1.451257152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:35.449671654 +0000 UTC m=+1682.374940629" watchObservedRunningTime="2026-10-31 00:42:35.451257152 +0000 UTC m=+1682.376526108" Oct 31 00:42:35.536096 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:35.536054 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:36.005566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:36.005496 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:42:36.006823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:36.006794 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:42:36.006961 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:36.006845 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:42:36.345111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:36.345031 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:42:36.541006 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:36.540973 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:37.005449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:37.005404 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:42:37.005667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:37.005480 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:42:37.345742 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:37.345660 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:37.346603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:37.346584 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:42:38.005962 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:38.005926 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:42:38.006377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:38.005984 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:42:38.443948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:38.443918 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:38.629397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:38.629363 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:38.793644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:38.793554 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:39.004868 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:39.004831 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:42:39.005040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:39.004888 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:42:39.729512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:39.729479 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:40.005586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:40.005471 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:42:40.005586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:40.005561 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 00:42:41.010477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:41.010445 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:42:41.011491 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:41.011472 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:42:42.099017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:42.098982 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:42.288049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:42.288008 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:43.184594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:43.184519 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:42:43.302902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:43.302863 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:42:43.542260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:43.542156 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:43.701353 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:43.701303 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.286458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:45.286427 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:48.185939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.185890 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" podUID="ae8935dc-7188-4a77-af06-016507fb84b9" containerName="payload-processing" containerID="cri-o://0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e" gracePeriod=25 Oct 31 00:42:48.303458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.303424 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" podUID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" containerName="payload-pre-processing" containerID="cri-o://1f390b002b6da8f6f4ca55ac0cb8f7b5334b4db0ff819042f20b992739891223" gracePeriod=25 Oct 31 00:42:48.463963 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.463936 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:42:48.498915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.498867 4471 generic.go:356] "Generic (PLEG): container finished" podID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" containerID="1f390b002b6da8f6f4ca55ac0cb8f7b5334b4db0ff819042f20b992739891223" exitCode=0 Oct 31 00:42:48.499088 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.498935 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" event={"ID":"5dac9294-8c4a-42fd-af87-fc4afc0a2f38","Type":"ContainerDied","Data":"1f390b002b6da8f6f4ca55ac0cb8f7b5334b4db0ff819042f20b992739891223"} Oct 31 00:42:48.500650 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.500625 4471 generic.go:356] "Generic (PLEG): container finished" podID="ae8935dc-7188-4a77-af06-016507fb84b9" containerID="0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e" exitCode=0 Oct 31 00:42:48.500780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.500688 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" Oct 31 00:42:48.500842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.500717 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" event={"ID":"ae8935dc-7188-4a77-af06-016507fb84b9","Type":"ContainerDied","Data":"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e"} Oct 31 00:42:48.500882 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.500872 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns" event={"ID":"ae8935dc-7188-4a77-af06-016507fb84b9","Type":"ContainerDied","Data":"0b341cacef31a9488c6802b997902e0cd3708c1138174ade6a8f44e778893abd"} Oct 31 00:42:48.500917 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.500907 4471 scope.go:122] "RemoveContainer" containerID="0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e" Oct 31 00:42:48.513449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.513425 4471 scope.go:122] "RemoveContainer" containerID="0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e" Oct 31 00:42:48.514028 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:42:48.513994 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e\": container with ID starting with 0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e not found: ID does not exist" containerID="0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e" Oct 31 00:42:48.514191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.514043 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e"} err="failed to get container status \"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e\": rpc error: code = NotFound desc = could not find container \"0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e\": container with ID starting with 0f3500d64f1e6fe8e9abca055f172c07b1a4f917d0972199bdc37b95cbc19e4e not found: ID does not exist" Oct 31 00:42:48.585641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.585614 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:42:48.622959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.622903 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\" (UniqueName: \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\") pod \"ae8935dc-7188-4a77-af06-016507fb84b9\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " Oct 31 00:42:48.623245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.623213 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\") pod \"ae8935dc-7188-4a77-af06-016507fb84b9\" (UID: \"ae8935dc-7188-4a77-af06-016507fb84b9\") " Oct 31 00:42:48.623815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.623784 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume" pod "ae8935dc-7188-4a77-af06-016507fb84b9" (UID: "ae8935dc-7188-4a77-af06-016507fb84b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:48.625520 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.625493 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw" pod "ae8935dc-7188-4a77-af06-016507fb84b9" (UID: "ae8935dc-7188-4a77-af06-016507fb84b9"). InnerVolumeSpecName "kube-api-access-chzgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:48.724671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.724576 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\") pod \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " Oct 31 00:42:48.724824 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.724796 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\" (UniqueName: \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\") pod \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\" (UID: \"5dac9294-8c4a-42fd-af87-fc4afc0a2f38\") " Oct 31 00:42:48.724961 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.724936 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume" pod "5dac9294-8c4a-42fd-af87-fc4afc0a2f38" (UID: "5dac9294-8c4a-42fd-af87-fc4afc0a2f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:48.725273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.725252 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.725273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.725271 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae8935dc-7188-4a77-af06-016507fb84b9-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.725421 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.725280 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzgw\" (UniqueName: \"kubernetes.io/projected/ae8935dc-7188-4a77-af06-016507fb84b9-kube-api-access-chzgw\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.726829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.726801 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9" pod "5dac9294-8c4a-42fd-af87-fc4afc0a2f38" (UID: "5dac9294-8c4a-42fd-af87-fc4afc0a2f38"). InnerVolumeSpecName "kube-api-access-m5sd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:48.826649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.826603 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5sd9\" (UniqueName: \"kubernetes.io/projected/5dac9294-8c4a-42fd-af87-fc4afc0a2f38-kube-api-access-m5sd9\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:48.961552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.955591 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:42:48.981606 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:48.976340 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-74c6ba99-cd559cfbf-v6nns"] Oct 31 00:42:49.211810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.211774 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8935dc-7188-4a77-af06-016507fb84b9" path="/var/lib/kubelet/pods/ae8935dc-7188-4a77-af06-016507fb84b9/volumes" Oct 31 00:42:49.284246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.284147 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:49.508853 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.508821 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" Oct 31 00:42:49.508853 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.508823 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc" event={"ID":"5dac9294-8c4a-42fd-af87-fc4afc0a2f38","Type":"ContainerDied","Data":"053460e84f91d7677fb6b561bca1d3041b12db2376c3a03298361c031381e8a1"} Oct 31 00:42:49.509124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.508903 4471 scope.go:122] "RemoveContainer" containerID="1f390b002b6da8f6f4ca55ac0cb8f7b5334b4db0ff819042f20b992739891223" Oct 31 00:42:49.692122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.692086 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:49.697651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.697506 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:42:49.697651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.697577 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-74c6ba99-794886fd59-cqrgc"] Oct 31 00:42:49.781252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781220 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:42:49.781863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781847 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ae8935dc-7188-4a77-af06-016507fb84b9" containerName="payload-processing" Oct 31 00:42:49.781905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781864 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8935dc-7188-4a77-af06-016507fb84b9" containerName="payload-processing" Oct 31 00:42:49.781905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781883 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" containerName="payload-pre-processing" Oct 31 00:42:49.781905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781889 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" containerName="payload-pre-processing" Oct 31 00:42:49.782013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.781998 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8935dc-7188-4a77-af06-016507fb84b9" containerName="payload-processing" Oct 31 00:42:49.782013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.782008 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" containerName="payload-pre-processing" Oct 31 00:42:49.785469 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.785452 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:49.849059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.849022 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7d5f4de7\"" Oct 31 00:42:49.849414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.849392 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7d5f4de7-dockercfg-s8cqn\"" Oct 31 00:42:49.850389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.850366 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:42:49.924552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.915258 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:42:49.924552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.918985 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:49.980603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.980156 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfr82\" (UniqueName: \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:49.980603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.980517 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.011565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:49.993094 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:42:50.100692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.100653 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfr82\" (UniqueName: \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.101089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.101066 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqcq\" (UniqueName: \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.101258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.101241 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.101402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.101388 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.116914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.113135 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.212617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.206716 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.212617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.206998 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqcq\" (UniqueName: \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.212617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.209494 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.221275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.221236 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfr82\" (UniqueName: \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\") pod \"payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.292032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.290251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqcq\" (UniqueName: \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\") pod \"payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.399148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.398574 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:50.417080 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.416810 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:50.829853 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:42:50.829815 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bc954d_59ed_4c85_9259_65b4898d0f61.slice/crio-67a8a65a1126fa40eb01c1766330e687a1eaf6142689b0cef2330c79edaea82b WatchSource:0}: Error finding container 67a8a65a1126fa40eb01c1766330e687a1eaf6142689b0cef2330c79edaea82b: Status 404 returned error can't find the container with id 67a8a65a1126fa40eb01c1766330e687a1eaf6142689b0cef2330c79edaea82b Oct 31 00:42:50.832165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.832141 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:42:50.889690 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:42:50.889655 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c04fb5_2756_4931_828d_904438781bfa.slice/crio-903e4633ffda37dba073657fa65ea1b1ea108d4cfda95277ae0d3090d92d8183 WatchSource:0}: Error finding container 903e4633ffda37dba073657fa65ea1b1ea108d4cfda95277ae0d3090d92d8183: Status 404 returned error can't find the container with id 903e4633ffda37dba073657fa65ea1b1ea108d4cfda95277ae0d3090d92d8183 Oct 31 00:42:50.892893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.892867 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:42:50.908396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:50.908366 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:51.216765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.216729 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dac9294-8c4a-42fd-af87-fc4afc0a2f38" path="/var/lib/kubelet/pods/5dac9294-8c4a-42fd-af87-fc4afc0a2f38/volumes" Oct 31 00:42:51.548325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.548239 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" event={"ID":"18c04fb5-2756-4931-828d-904438781bfa","Type":"ContainerStarted","Data":"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc"} Oct 31 00:42:51.548325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.548291 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" event={"ID":"18c04fb5-2756-4931-828d-904438781bfa","Type":"ContainerStarted","Data":"903e4633ffda37dba073657fa65ea1b1ea108d4cfda95277ae0d3090d92d8183"} Oct 31 00:42:51.548598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.548349 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:42:51.549861 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.549840 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" event={"ID":"30bc954d-59ed-4c85-9259-65b4898d0f61","Type":"ContainerStarted","Data":"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b"} Oct 31 00:42:51.549940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.549874 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" event={"ID":"30bc954d-59ed-4c85-9259-65b4898d0f61","Type":"ContainerStarted","Data":"67a8a65a1126fa40eb01c1766330e687a1eaf6142689b0cef2330c79edaea82b"} Oct 31 00:42:51.550385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.550369 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:42:51.697554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.696594 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" podStartSLOduration=2.696571775 podStartE2EDuration="2.696571775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:51.694585176 +0000 UTC m=+1698.619854129" watchObservedRunningTime="2026-10-31 00:42:51.696571775 +0000 UTC m=+1698.621840729" Oct 31 00:42:51.844356 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:51.844199 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" podStartSLOduration=2.844185673 podStartE2EDuration="2.844185673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:51.842856138 +0000 UTC m=+1698.768125090" watchObservedRunningTime="2026-10-31 00:42:51.844185673 +0000 UTC m=+1698.769454624" Oct 31 00:42:56.050325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:56.050289 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:56.208041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:56.208003 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:57.739760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:57.739713 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:57.740159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:57.739786 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:59.655908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.655872 4471 generic.go:356] "Generic (PLEG): container finished" podID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerID="004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258" exitCode=0 Oct 31 00:42:59.656397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.655944 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerDied","Data":"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258"} Oct 31 00:42:59.656488 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.656470 4471 scope.go:122] "RemoveContainer" containerID="004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258" Oct 31 00:42:59.658329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658309 4471 generic.go:356] "Generic (PLEG): container finished" podID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerID="8c178d33aa86580391153f1a4bbf0d7e25806b61578011d399e75b3afc4473c6" exitCode=0 Oct 31 00:42:59.658437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658340 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerDied","Data":"8c178d33aa86580391153f1a4bbf0d7e25806b61578011d399e75b3afc4473c6"} Oct 31 00:42:59.658600 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658569 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:59.658679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658663 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:59.658727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658713 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:59.658760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.658726 4471 scope.go:122] "RemoveContainer" containerID="8c178d33aa86580391153f1a4bbf0d7e25806b61578011d399e75b3afc4473c6" Oct 31 00:42:59.660227 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.660201 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:59.660320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.660267 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:42:59.660320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:42:59.660302 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:00.711818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:00.711574 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerStarted","Data":"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7"} Oct 31 00:43:00.714563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:00.714506 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerStarted","Data":"081f17dd135c866747e8e3640249c00526f54c0aa4c1ab974bddec54564b9191"} Oct 31 00:43:01.223251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.223197 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:01.621314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.621226 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:43:01.621643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.621345 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:43:01.621834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.621812 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:43:01.687555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.684377 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:43:01.687555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.684513 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:43:01.687555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:01.684586 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:43:02.494553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.493780 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:02.605848 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.605813 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:43:02.621515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.621484 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:43:02.621704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.621562 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:43:02.683554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.682953 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:43:02.683554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:02.683035 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:43:03.611601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:03.611567 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:43:03.621829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:03.621801 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:43:03.621980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:03.621854 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:43:03.689150 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:03.689102 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:43:03.689463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:03.689414 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:43:04.332405 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:04.332370 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:43:04.622029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:04.621946 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:43:04.622029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:04.622006 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:43:04.684554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:04.682141 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:43:04.684554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:04.682205 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:43:05.621583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:05.621547 4471 patch_prober.go:29] interesting pod/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 00:43:05.621773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:05.621605 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 00:43:05.685554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:05.685102 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:43:05.685554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:05.685164 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:43:06.199985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.199944 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:06.625669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.625593 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:43:06.625902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.625879 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:43:06.626643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.626621 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:43:06.686732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.686697 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:43:06.687126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.687003 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:43:06.687767 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:06.687745 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 00:43:10.689798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:10.689762 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.492736 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:12.492682 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:12.912408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:12.912371 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:13.340257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:13.340162 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:14.508694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:14.508659 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:43:14.509131 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:14.508848 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" containerID="cri-o://edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8" gracePeriod=30 Oct 31 00:43:15.087252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:15.087213 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:15.908995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:15.908962 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:17.861702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:17.861663 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.215185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.215153 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:43:18.219661 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.219636 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.278509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.278472 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-f32ec2-openshift-default-dockercfg-6lngs\"" Oct 31 00:43:18.311281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.311220 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.330483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.330449 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:43:18.371777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.371738 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.371949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.371791 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.371949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.371842 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372131 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372194 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372516 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372496 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372614 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmhnq\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372651 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.372805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.372791 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.473952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.473852 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.473952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.473891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.473952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.473930 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.473982 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmhnq\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474007 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474084 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474116 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474148 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474170 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.474519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.474403 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.475088 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.475059 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:18.475376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.475181 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.475569 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.475524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.475706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.475688 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.476079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.476050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.478385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.478363 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.479017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.478995 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.536391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.536350 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.536569 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.536453 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmhnq\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\") pod \"e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:18.835450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:18.835336 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:19.264963 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:43:19.264926 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe7a287_b617_42f4_b531_a891fe920719.slice/crio-afec8ab83d602f5d70c09d49c46164672bc74e4a1c4028f586a589ba8d7ebd58 WatchSource:0}: Error finding container afec8ab83d602f5d70c09d49c46164672bc74e4a1c4028f586a589ba8d7ebd58: Status 404 returned error can't find the container with id afec8ab83d602f5d70c09d49c46164672bc74e4a1c4028f586a589ba8d7ebd58 Oct 31 00:43:19.267899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.267862 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:19.267999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.267935 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:19.267999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.267966 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:19.268075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.268054 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:43:19.786037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.785966 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:43:19.849360 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.849324 4471 generic.go:356] "Generic (PLEG): container finished" podID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerID="edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8" exitCode=0 Oct 31 00:43:19.849566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.849378 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" Oct 31 00:43:19.849566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.849403 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" event={"ID":"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e","Type":"ContainerDied","Data":"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8"} Oct 31 00:43:19.849566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.849478 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7" event={"ID":"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e","Type":"ContainerDied","Data":"4e60a3d2ff6e8ffcd09b36b89fc07105b111c75dae9caab27df1de6d4f325888"} Oct 31 00:43:19.849566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.849507 4471 scope.go:122] "RemoveContainer" containerID="edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8" Oct 31 00:43:19.851150 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.851094 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerStarted","Data":"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89"} Oct 31 00:43:19.851269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.851166 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerStarted","Data":"afec8ab83d602f5d70c09d49c46164672bc74e4a1c4028f586a589ba8d7ebd58"} Oct 31 00:43:19.859729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.859709 4471 scope.go:122] "RemoveContainer" containerID="edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8" Oct 31 00:43:19.859982 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:19.859964 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8\": container with ID starting with edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8 not found: ID does not exist" containerID="edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8" Oct 31 00:43:19.860016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.859992 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8"} err="failed to get container status \"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8\": rpc error: code = NotFound desc = could not find container \"edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8\": container with ID starting with edabb5979cb0a703257e76b166119e9d1325ba5d039d4c24cb0a715c8016dcc8 not found: ID does not exist" Oct 31 00:43:19.894378 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894347 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894485 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894518 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894584 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894602 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894657 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894623 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:19.894807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894693 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894755 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894807 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894799 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894854 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\") pod \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\" (UID: \"12e7f5da-c5cd-48be-82ea-ff89e53d6e3e\") " Oct 31 00:43:19.894959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894851 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:19.894959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894920 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:19.895116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.894980 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:19.895301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895282 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:19.895575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895558 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.895575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895576 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.895721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895585 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.895721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895594 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.895721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.895602 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.897000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.896959 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:19.897000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.896968 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:19.897165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.897148 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:19.897224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.897178 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d" pod "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" (UID: "12e7f5da-c5cd-48be-82ea-ff89e53d6e3e"). InnerVolumeSpecName "kube-api-access-n4r9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:19.997673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.997641 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.997673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.997671 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4r9d\" (UniqueName: \"kubernetes.io/projected/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-kube-api-access-n4r9d\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.997673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.997681 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:19.998018 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:19.997691 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:20.040039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.039929 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podStartSLOduration=2.039907165 podStartE2EDuration="2.039907165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:20.037433847 +0000 UTC m=+1726.962702821" watchObservedRunningTime="2026-10-31 00:43:20.039907165 +0000 UTC m=+1726.965176119" Oct 31 00:43:20.288125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.288038 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:43:20.291907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.291793 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-4r6d7"] Oct 31 00:43:20.836418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.836387 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:20.842956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.842926 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:20.858885 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.858847 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:20.859772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:20.859748 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:43:21.212888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:21.212847 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" path="/var/lib/kubelet/pods/12e7f5da-c5cd-48be-82ea-ff89e53d6e3e/volumes" Oct 31 00:43:22.338551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:22.338421 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.457394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:23.457357 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.685819 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:23.685784 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:23.909410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:23.909370 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.469959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:24.469927 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.646953 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:24.646918 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:24.808426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:24.808317 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.024372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.024341 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.050177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.050128 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:43:25.050431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.050393 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" containerID="cri-o://15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f" gracePeriod=30 Oct 31 00:43:25.264593 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.250646 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.665547 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.665499 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:25.889352 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:25.889307 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.013328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:26.013233 4471 patch_prober.go:29] interesting pod/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 00:43:26.013328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:26.013311 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:26.082729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:26.082690 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:26.297831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:26.297747 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:27.190618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:27.190579 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:27.385970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:27.385935 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.629339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:28.629292 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:29.001795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:29.001761 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:29.519877 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:29.519847 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:43:29.520087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:29.520064 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" containerID="cri-o://a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4" gracePeriod=30 Oct 31 00:43:30.285219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.285116 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:30.382428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.382396 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:43:30.418315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418280 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418606 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418762 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418788 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418867 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418910 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418940 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.418980 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.419874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.419756 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.419874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.419759 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.419874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.419802 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420061 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420140 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\") pod \"277be621-e3ed-4388-b43f-6bde98ebcf50\" (UID: \"277be621-e3ed-4388-b43f-6bde98ebcf50\") " Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420462 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420851 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420873 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420891 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277be621-e3ed-4388-b43f-6bde98ebcf50-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420904 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.420939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.420915 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.422611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.422508 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:30.429276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.429239 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:30.429433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.429244 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "kube-api-access-2kw8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:30.430997 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.430954 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token" pod "277be621-e3ed-4388-b43f-6bde98ebcf50" (UID: "277be621-e3ed-4388-b43f-6bde98ebcf50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:30.451071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.451041 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:30.522993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.522955 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.522993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.522994 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kw8s\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-kube-api-access-2kw8s\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.523191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.523009 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.523191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.523023 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277be621-e3ed-4388-b43f-6bde98ebcf50-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:30.606211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.606123 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:30.962075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.962033 4471 generic.go:356] "Generic (PLEG): container finished" podID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerID="15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f" exitCode=0 Oct 31 00:43:30.962286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.962103 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" Oct 31 00:43:30.962286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.962243 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerDied","Data":"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f"} Oct 31 00:43:30.962389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.962296 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn" event={"ID":"277be621-e3ed-4388-b43f-6bde98ebcf50","Type":"ContainerDied","Data":"173589bd3ac5800cd47bdf07200f40a5521a2ff492312d3679144958bf251fab"} Oct 31 00:43:30.962389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.962326 4471 scope.go:122] "RemoveContainer" containerID="15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f" Oct 31 00:43:30.976415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.976387 4471 scope.go:122] "RemoveContainer" containerID="76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5" Oct 31 00:43:30.986479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.986450 4471 scope.go:122] "RemoveContainer" containerID="15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f" Oct 31 00:43:30.986814 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:30.986790 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f\": container with ID starting with 15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f not found: ID does not exist" containerID="15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f" Oct 31 00:43:30.986868 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.986824 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f"} err="failed to get container status \"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f\": rpc error: code = NotFound desc = could not find container \"15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f\": container with ID starting with 15474348dfc6724e4d92d63801ffde56d975c72746ac9a0a70654657f1c3ec0f not found: ID does not exist" Oct 31 00:43:30.986868 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.986844 4471 scope.go:122] "RemoveContainer" containerID="76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5" Oct 31 00:43:30.987135 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:30.987113 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5\": container with ID starting with 76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5 not found: ID does not exist" containerID="76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5" Oct 31 00:43:30.987173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:30.987144 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5"} err="failed to get container status \"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5\": rpc error: code = NotFound desc = could not find container \"76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5\": container with ID starting with 76e7ede64d490a4531570cdb83ca9df5bbf8e1cb9bad04e083cd55fcc97817c5 not found: ID does not exist" Oct 31 00:43:31.132487 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.132456 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:43:31.182237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.182202 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-74c6ba99-openshift-default-65cd5b8f85-8hddn"] Oct 31 00:43:31.209754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.209718 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" path="/var/lib/kubelet/pods/277be621-e3ed-4388-b43f-6bde98ebcf50/volumes" Oct 31 00:43:31.602224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.602191 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.794338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.794298 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:31.975480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:31.975449 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.513098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:32.513062 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:32.693551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:32.693323 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.202057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:33.202016 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:33.520342 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:33.520241 4471 scope.go:122] "RemoveContainer" containerID="5d7601661e8c3c2eb9c80f83131bdd16cfea46224b42ae906deb12244140e2a4" Oct 31 00:43:34.283394 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:34.283359 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:43:34.669865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.669833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:34.787971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.787946 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:43:34.908153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908115 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908153 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908176 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908244 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908278 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908310 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908451 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908640 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.908826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908445 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:34.908826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908740 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:34.908826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908773 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\") pod \"ef6e4f0d-30df-4293-a749-93f0de3fed63\" (UID: \"ef6e4f0d-30df-4293-a749-93f0de3fed63\") " Oct 31 00:43:34.909301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908929 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:34.909301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.908969 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:34.909301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.909134 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:34.910262 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.910236 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:34.910367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.910271 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:34.910367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.910287 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:34.910367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.910314 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6e4f0d-30df-4293-a749-93f0de3fed63-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:34.910367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.910328 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:34.922387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.919512 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:34.922859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.922825 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:43:34.923023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.922582 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:34.923023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.922916 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "kube-api-access-lk84l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:34.923330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.923309 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo" pod "ef6e4f0d-30df-4293-a749-93f0de3fed63" (UID: "ef6e4f0d-30df-4293-a749-93f0de3fed63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:43:34.992430 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.992390 4471 generic.go:356] "Generic (PLEG): container finished" podID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerID="a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4" exitCode=0 Oct 31 00:43:34.992658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.992459 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" Oct 31 00:43:34.992658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.992483 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" event={"ID":"ef6e4f0d-30df-4293-a749-93f0de3fed63","Type":"ContainerDied","Data":"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4"} Oct 31 00:43:34.992658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.992590 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk" event={"ID":"ef6e4f0d-30df-4293-a749-93f0de3fed63","Type":"ContainerDied","Data":"b25c3395a6f5d2c2e8ea3acd8237a6ca069ba7a40919ef1f636cbc84e02e6993"} Oct 31 00:43:34.992658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:34.992617 4471 scope.go:122] "RemoveContainer" containerID="a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4" Oct 31 00:43:35.006217 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.006200 4471 scope.go:122] "RemoveContainer" containerID="a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4" Oct 31 00:43:35.006595 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:43:35.006567 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4\": container with ID starting with a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4 not found: ID does not exist" containerID="a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4" Oct 31 00:43:35.006709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.006608 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4"} err="failed to get container status \"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4\": rpc error: code = NotFound desc = could not find container \"a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4\": container with ID starting with a06e295bf09bd9ad772f36c801f12e11b76d616d1bdb3592119b5ba246dd8ed4 not found: ID does not exist" Oct 31 00:43:35.011515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.011488 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk84l\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-kube-api-access-lk84l\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:35.011679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.011516 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:35.011679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.011561 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:35.011679 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.011574 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6e4f0d-30df-4293-a749-93f0de3fed63-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:35.128580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.114313 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:35.165282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.165045 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:43:35.171018 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.170987 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-5dxxk"] Oct 31 00:43:35.213256 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:35.213223 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" path="/var/lib/kubelet/pods/ef6e4f0d-30df-4293-a749-93f0de3fed63/volumes" Oct 31 00:43:36.247310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:36.247274 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:39.541358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:39.541311 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:39.705872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:39.705836 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.368084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:40.368040 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:40.569019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:40.568944 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.433122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:41.433092 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.598298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:41.598253 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:41.804027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:41.803935 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:46.273631 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.273542 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:43:46.274396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274366 4471 cpu_manager.go:414] "containerMap: removing container" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:43:46.274396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274395 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:43:46.274622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274414 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" Oct 31 00:43:46.274622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274422 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" Oct 31 00:43:46.274622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274459 4471 cpu_manager.go:414] "containerMap: removing container" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" Oct 31 00:43:46.274622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274468 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" Oct 31 00:43:46.274812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274657 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:43:46.274812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274677 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6e4f0d-30df-4293-a749-93f0de3fed63" containerName="istio-proxy" Oct 31 00:43:46.274812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274689 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:43:46.274812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.274700 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e7f5da-c5cd-48be-82ea-ff89e53d6e3e" containerName="istio-proxy" Oct 31 00:43:46.278236 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.278214 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.332497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.332465 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:43:46.332671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.332655 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:43:46.381977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.381945 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:43:46.382452 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.382190 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pbwgm\"" Oct 31 00:43:46.387311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.387285 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:43:46.400002 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.399972 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtw6\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400032 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400092 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400263 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400285 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400301 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.400397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.400376 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.501834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.501800 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.501834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.501838 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.501860 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.501897 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.501968 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtw6\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.502008 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.502027 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.502872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.502832 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.504448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.504417 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.504616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.504449 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.504697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.504669 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.504841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.504821 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.567148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.567077 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.567354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.567334 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtw6\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-btpxb\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:46.590382 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:46.590354 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:47.045607 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:43:47.045568 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f21c62_f7c3_416c_89b7_ab2ee2302f5a.slice/crio-8e10d0adf8c44e3de541f97a9b625f0c69172c17e83027dffa168e8b44680c4b WatchSource:0}: Error finding container 8e10d0adf8c44e3de541f97a9b625f0c69172c17e83027dffa168e8b44680c4b: Status 404 returned error can't find the container with id 8e10d0adf8c44e3de541f97a9b625f0c69172c17e83027dffa168e8b44680c4b Oct 31 00:43:47.047844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.047824 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:43:47.049416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.049396 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:43:47.096955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.096916 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" event={"ID":"81f21c62-f7c3-416c-89b7-ab2ee2302f5a","Type":"ContainerStarted","Data":"8e10d0adf8c44e3de541f97a9b625f0c69172c17e83027dffa168e8b44680c4b"} Oct 31 00:43:47.220978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.220945 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.552192 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.552107 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:47.725623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:47.725588 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:51.402245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:51.402212 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:51.402504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:51.402281 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:43:52.187277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.187236 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" event={"ID":"81f21c62-f7c3-416c-89b7-ab2ee2302f5a","Type":"ContainerStarted","Data":"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e"} Oct 31 00:43:52.187477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.187453 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:52.189767 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.189742 4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-btpxb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:43:52.189853 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.189787 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:52.359584 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.359516 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" podStartSLOduration=2.006714853 podStartE2EDuration="6.35950291s" totalImagesPullingTime="4.352788057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:52.307759497 +0000 UTC m=+1759.233028450" watchObservedRunningTime="2026-10-31 00:43:52.35950291 +0000 UTC m=+1759.284771862" Oct 31 00:43:52.740329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:52.740292 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:53.194124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:53.194086 4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-btpxb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:43:53.194290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:53.194142 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:43:56.194140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:56.194106 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:43:56.534667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:56.534500 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:43:56.662437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:56.662398 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:43:56.947451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:56.947380 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.134739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:57.134706 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:58.154474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:43:58.154432 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.403660 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.403615 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:44:01.404145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.403841 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" containerID="cri-o://3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091" gracePeriod=30 Oct 31 00:44:01.534946 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.534907 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" podUID="24b5dec0-e590-4809-b749-e25802054e92" containerName="payload-processing" containerID="cri-o://9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4" gracePeriod=25 Oct 31 00:44:01.663544 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.663484 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" podUID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" containerName="payload-pre-processing" containerID="cri-o://09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2" gracePeriod=25 Oct 31 00:44:01.812838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.812802 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:44:01.903436 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.903400 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\" (UniqueName: \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\") pod \"24b5dec0-e590-4809-b749-e25802054e92\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " Oct 31 00:44:01.903667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.903641 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\") pod \"24b5dec0-e590-4809-b749-e25802054e92\" (UID: \"24b5dec0-e590-4809-b749-e25802054e92\") " Oct 31 00:44:01.904148 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.904078 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume" pod "24b5dec0-e590-4809-b749-e25802054e92" (UID: "24b5dec0-e590-4809-b749-e25802054e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:01.904449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.904426 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b5dec0-e590-4809-b749-e25802054e92-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.905749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.905720 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp" pod "24b5dec0-e590-4809-b749-e25802054e92" (UID: "24b5dec0-e590-4809-b749-e25802054e92"). InnerVolumeSpecName "kube-api-access-tpxkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:01.925101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:01.925079 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:44:02.005233 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.005199 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\" (UniqueName: \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\") pod \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " Oct 31 00:44:02.005402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.005379 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\") pod \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\" (UID: \"dc528a86-c6b5-4664-a43c-f1016fa1aff9\") " Oct 31 00:44:02.005753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.005729 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxkp\" (UniqueName: \"kubernetes.io/projected/24b5dec0-e590-4809-b749-e25802054e92-kube-api-access-tpxkp\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:02.005865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.005770 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume" pod "dc528a86-c6b5-4664-a43c-f1016fa1aff9" (UID: "dc528a86-c6b5-4664-a43c-f1016fa1aff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:02.007320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.007298 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d" pod "dc528a86-c6b5-4664-a43c-f1016fa1aff9" (UID: "dc528a86-c6b5-4664-a43c-f1016fa1aff9"). InnerVolumeSpecName "kube-api-access-zz98d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:02.107164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.107126 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc528a86-c6b5-4664-a43c-f1016fa1aff9-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:02.107164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.107158 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz98d\" (UniqueName: \"kubernetes.io/projected/dc528a86-c6b5-4664-a43c-f1016fa1aff9-kube-api-access-zz98d\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:02.248244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.248154 4471 generic.go:356] "Generic (PLEG): container finished" podID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" containerID="09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2" exitCode=0 Oct 31 00:44:02.248244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.248207 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" Oct 31 00:44:02.248471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.248247 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" event={"ID":"dc528a86-c6b5-4664-a43c-f1016fa1aff9","Type":"ContainerDied","Data":"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2"} Oct 31 00:44:02.248471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.248301 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g" event={"ID":"dc528a86-c6b5-4664-a43c-f1016fa1aff9","Type":"ContainerDied","Data":"7307b21ac60eaa54b0af5cc46d6fe2c105920031077ac4211c6662f6a406ab24"} Oct 31 00:44:02.248471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.248326 4471 scope.go:122] "RemoveContainer" containerID="09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2" Oct 31 00:44:02.249777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.249743 4471 generic.go:356] "Generic (PLEG): container finished" podID="24b5dec0-e590-4809-b749-e25802054e92" containerID="9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4" exitCode=0 Oct 31 00:44:02.249907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.249794 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" Oct 31 00:44:02.249907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.249827 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" event={"ID":"24b5dec0-e590-4809-b749-e25802054e92","Type":"ContainerDied","Data":"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4"} Oct 31 00:44:02.249907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.249865 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675" event={"ID":"24b5dec0-e590-4809-b749-e25802054e92","Type":"ContainerDied","Data":"abf31ecdce4a9e167ad1de589912f4f39567242366c7ff27e4a590e3e4c16efc"} Oct 31 00:44:02.259613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.259594 4471 scope.go:122] "RemoveContainer" containerID="09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2" Oct 31 00:44:02.259920 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:44:02.259892 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2\": container with ID starting with 09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2 not found: ID does not exist" containerID="09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2" Oct 31 00:44:02.260042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.259929 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2"} err="failed to get container status \"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2\": rpc error: code = NotFound desc = could not find container \"09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2\": container with ID starting with 09c7606dcabe7f8468a682c642264139d0ca633edf2f380750681f7e9ead4bb2 not found: ID does not exist" Oct 31 00:44:02.260042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.259954 4471 scope.go:122] "RemoveContainer" containerID="9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4" Oct 31 00:44:02.269953 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.269936 4471 scope.go:122] "RemoveContainer" containerID="9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4" Oct 31 00:44:02.270218 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:44:02.270200 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4\": container with ID starting with 9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4 not found: ID does not exist" containerID="9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4" Oct 31 00:44:02.270266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.270226 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4"} err="failed to get container status \"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4\": rpc error: code = NotFound desc = could not find container \"9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4\": container with ID starting with 9412e47086aed5f937f00b6488268f051ab8807086e4c6b87ab8df35eca502a4 not found: ID does not exist" Oct 31 00:44:02.435504 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.435030 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:44:02.467164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.467130 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f0651cac-6dbbdb5867-fdt2g"] Oct 31 00:44:02.597774 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.597743 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:44:02.602647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:02.602624 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f0651cac-59bc6987b9-mh675"] Oct 31 00:44:03.209789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:03.209756 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b5dec0-e590-4809-b749-e25802054e92" path="/var/lib/kubelet/pods/24b5dec0-e590-4809-b749-e25802054e92/volumes" Oct 31 00:44:03.210158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:03.210145 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" path="/var/lib/kubelet/pods/dc528a86-c6b5-4664-a43c-f1016fa1aff9/volumes" Oct 31 00:44:03.214879 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:03.214836 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:04.282063 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:44:04.282026 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:44:05.062428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:05.062393 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:06.677778 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.677708 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:44:06.781952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.781917 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782053 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782110 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782147 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782241 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.782298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782250 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782409 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782362 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782462 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782399 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782462 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782454 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782487 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.782614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782518 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\") pod \"d7184561-f832-47f3-aed5-5e6a44f2d18f\" (UID: \"d7184561-f832-47f3-aed5-5e6a44f2d18f\") " Oct 31 00:44:06.782715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782660 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:06.782715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782643 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.782805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.782719 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.783198 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.783173 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.783198 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.783196 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.783398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.783207 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7184561-f832-47f3-aed5-5e6a44f2d18f-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.783398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.783215 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.783398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.783225 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.784455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.784433 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:06.784780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.784757 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "kube-api-access-b97f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:06.784840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.784793 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:06.784876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.784838 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy" pod "d7184561-f832-47f3-aed5-5e6a44f2d18f" (UID: "d7184561-f832-47f3-aed5-5e6a44f2d18f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:06.884296 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.884257 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.884296 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.884289 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.884296 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.884299 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97f8\" (UniqueName: \"kubernetes.io/projected/d7184561-f832-47f3-aed5-5e6a44f2d18f-kube-api-access-b97f8\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.884551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:06.884310 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7184561-f832-47f3-aed5-5e6a44f2d18f-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:07.306266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.306227 4471 generic.go:356] "Generic (PLEG): container finished" podID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerID="3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091" exitCode=0 Oct 31 00:44:07.306454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.306275 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" Oct 31 00:44:07.306454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.306310 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" event={"ID":"d7184561-f832-47f3-aed5-5e6a44f2d18f","Type":"ContainerDied","Data":"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091"} Oct 31 00:44:07.306454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.306368 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7" event={"ID":"d7184561-f832-47f3-aed5-5e6a44f2d18f","Type":"ContainerDied","Data":"24b836bf687849f19cd31f5870281b6c9bf684032ea11f7810728f54a5ecef97"} Oct 31 00:44:07.306454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.306392 4471 scope.go:122] "RemoveContainer" containerID="3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091" Oct 31 00:44:07.316425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.316405 4471 scope.go:122] "RemoveContainer" containerID="3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091" Oct 31 00:44:07.316701 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:44:07.316683 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091\": container with ID starting with 3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091 not found: ID does not exist" containerID="3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091" Oct 31 00:44:07.316757 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.316711 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091"} err="failed to get container status \"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091\": rpc error: code = NotFound desc = could not find container \"3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091\": container with ID starting with 3d96e41e8b5fb2ea824321bb83b0241dd490f66edd6ee01d259fec5339c25091 not found: ID does not exist" Oct 31 00:44:07.473672 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.473592 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:44:07.523306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:07.523274 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-dq5s7"] Oct 31 00:44:08.194497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:08.194462 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:09.209268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:09.209236 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" path="/var/lib/kubelet/pods/d7184561-f832-47f3-aed5-5e6a44f2d18f/volumes" Oct 31 00:44:11.616732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:11.616694 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:13.171478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:13.171441 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:18.086313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:18.086275 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:22.997495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:22.997461 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:25.945973 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:25.945930 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:26.842064 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:26.842031 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.296131 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.296047 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:27.507976 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.507941 4471 generic.go:356] "Generic (PLEG): container finished" podID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerID="f1b58f2b6105866ec3bd2c124610f61131997d742f96f53ef033b8e3eb80cedf" exitCode=0 Oct 31 00:44:27.508171 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.508122 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerDied","Data":"f1b58f2b6105866ec3bd2c124610f61131997d742f96f53ef033b8e3eb80cedf"} Oct 31 00:44:27.508564 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.508525 4471 scope.go:122] "RemoveContainer" containerID="f1b58f2b6105866ec3bd2c124610f61131997d742f96f53ef033b8e3eb80cedf" Oct 31 00:44:27.510318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.510284 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:27.510412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.510369 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:27.510454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.510412 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:44:27.981634 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:27.980303 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.137182 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.137131 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.296622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.296441 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:28.518456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.518423 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerStarted","Data":"343658fd226d56679218b45c50a0698a9852842e7077fa60cdb8483f494b40ec"} Oct 31 00:44:28.535926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.535892 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:44:28.536219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.536197 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:28.536336 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:28.536259 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:29.536380 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:29.536338 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:29.536928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:29.536398 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:30.535392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:30.535359 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:30.535611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:30.535423 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:31.536403 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:31.536369 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:31.536863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:31.536421 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:31.807049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:31.806966 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.042834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:32.042798 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.221173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:32.221138 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:32.536301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:32.536212 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:32.536301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:32.536273 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:32.808792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:32.808700 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:33.536169 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:33.536127 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:44:33.536380 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:33.536195 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 00:44:34.281735 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:44:34.281700 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:44:34.540817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:34.540740 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:44:34.541070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:34.541042 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:44:34.541925 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:34.541905 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:44:34.657888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:34.657853 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:34.855851 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:34.855723 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:35.011957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:35.011927 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:35.951550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:35.951492 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.291211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:36.291122 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:37.036827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:37.036790 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.648206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:38.648171 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:38.893713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:38.893679 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.314890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:40.314849 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.470097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:40.470066 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:40.620321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:40.620235 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:43.191939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:43.191902 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:43.349499 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:43.349458 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:43.536802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:43.536715 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:43.546411 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:43.546374 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:44:43.546674 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:43.546646 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" containerID="cri-o://081f17dd135c866747e8e3640249c00526f54c0aa4c1ab974bddec54564b9191" gracePeriod=30 Oct 31 00:44:45.693176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:45.693144 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.446383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.446344 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:48.686686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.686653 4471 generic.go:356] "Generic (PLEG): container finished" podID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerID="081f17dd135c866747e8e3640249c00526f54c0aa4c1ab974bddec54564b9191" exitCode=0 Oct 31 00:44:48.686860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.686721 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerDied","Data":"081f17dd135c866747e8e3640249c00526f54c0aa4c1ab974bddec54564b9191"} Oct 31 00:44:48.686860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.686784 4471 scope.go:122] "RemoveContainer" containerID="8c178d33aa86580391153f1a4bbf0d7e25806b61578011d399e75b3afc4473c6" Oct 31 00:44:48.832716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.832687 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:44:48.937743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.937702 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.937944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.937782 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.937944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.937850 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.937944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.937883 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.937992 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938052 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938122 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938170 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938236 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:48.938359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938345 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\") pod \"f6de6233-4820-4d44-84a5-b6d90b273a46\" (UID: \"f6de6233-4820-4d44-84a5-b6d90b273a46\") " Oct 31 00:44:48.938420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938391 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:48.938552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938517 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:48.938911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938849 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:48.938911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.938882 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:48.939289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.939027 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:48.939289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.939049 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:48.939289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.939064 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:48.939289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.939078 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:48.939289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.939092 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6de6233-4820-4d44-84a5-b6d90b273a46-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:48.940613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.940587 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:44:48.940949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.940923 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:44:48.941156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.941130 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "kube-api-access-jg4cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:48.941511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:48.941493 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token" pod "f6de6233-4820-4d44-84a5-b6d90b273a46" (UID: "f6de6233-4820-4d44-84a5-b6d90b273a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:49.040957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.040922 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:49.040957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.040952 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:49.040957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.040965 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:49.041192 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.040978 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4cs\" (UniqueName: \"kubernetes.io/projected/f6de6233-4820-4d44-84a5-b6d90b273a46-kube-api-access-jg4cs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:49.180245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.180209 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:49.373388 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.373348 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:49.696320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.696290 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" Oct 31 00:44:49.696822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.696289 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv" event={"ID":"f6de6233-4820-4d44-84a5-b6d90b273a46","Type":"ContainerDied","Data":"7072bc1b9c91954c084c4a0f0046ff48c4ef4326da0545acf03f9ab111ace0a8"} Oct 31 00:44:49.696822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.696433 4471 scope.go:122] "RemoveContainer" containerID="081f17dd135c866747e8e3640249c00526f54c0aa4c1ab974bddec54564b9191" Oct 31 00:44:49.867545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.867498 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:44:49.876054 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:49.876025 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f0651cac-openshift-default-7fc6f9555c-pxtlv"] Oct 31 00:44:50.766596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:50.766560 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:51.209708 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:51.209674 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" path="/var/lib/kubelet/pods/f6de6233-4820-4d44-84a5-b6d90b273a46/volumes" Oct 31 00:44:54.118790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:54.118752 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.340897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:54.340867 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:54.492347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:54.492313 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.044229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:56.044192 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.279325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:56.279283 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:56.522104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:56.522072 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.196555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:57.189412 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.369020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:57.368982 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:57.544925 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:44:57.544782 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.202443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.202406 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:00.202965 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.202950 4471 cpu_manager.go:414] "containerMap: removing container" podUID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" containerName="payload-pre-processing" Oct 31 00:45:00.203007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.202968 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" containerName="payload-pre-processing" Oct 31 00:45:00.203007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.202991 4471 cpu_manager.go:414] "containerMap: removing container" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:45:00.203007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.202996 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277be621-e3ed-4388-b43f-6bde98ebcf50" containerName="istio-proxy" Oct 31 00:45:00.203007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203005 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203010 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203025 4471 cpu_manager.go:414] "containerMap: removing container" podUID="24b5dec0-e590-4809-b749-e25802054e92" containerName="payload-processing" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203029 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b5dec0-e590-4809-b749-e25802054e92" containerName="payload-processing" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203048 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203052 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203069 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203074 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203156 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203166 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6de6233-4820-4d44-84a5-b6d90b273a46" containerName="istio-proxy" Oct 31 00:45:00.203317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203172 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc528a86-c6b5-4664-a43c-f1016fa1aff9" containerName="payload-pre-processing" Oct 31 00:45:00.203317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203181 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7184561-f832-47f3-aed5-5e6a44f2d18f" containerName="istio-proxy" Oct 31 00:45:00.203317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.203188 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b5dec0-e590-4809-b749-e25802054e92" containerName="payload-processing" Oct 31 00:45:00.206703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.206687 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.246106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.246055 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:00.250216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.250194 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:00.250366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.250308 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.258516 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.257217 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:00.259249 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.259221 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.263836 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.263813 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.275640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.275604 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.275896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.275876 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx7q\" (UniqueName: \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.276016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.276001 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.276249 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.276226 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgcww\" (UniqueName: \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.276590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.276377 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.276590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.276477 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsdv5\" (UniqueName: \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.299910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.299862 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:00.304836 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.304808 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:00.305022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.304998 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.309561 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.309504 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:00.342983 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.341371 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:00.354637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.352728 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 00:45:00.354637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.354357 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:00.358624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.358601 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:00.378368 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378334 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.378522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378383 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.378639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378600 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfplf\" (UniqueName: \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.378706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378647 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx7q\" (UniqueName: \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.378783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378757 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.378843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378819 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgcww\" (UniqueName: \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.378894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378851 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.378952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.379024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378972 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.379024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.378992 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2b2\" (UniqueName: \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.379116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.379061 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsdv5\" (UniqueName: \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.379728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.379706 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.379861 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.379840 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.448483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.448445 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgcww\" (UniqueName: \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.448670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.448446 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx7q\" (UniqueName: \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.450854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.450825 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsdv5\" (UniqueName: \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.480582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.480492 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.480720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.480626 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfplf\" (UniqueName: \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.480769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.480743 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.480878 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.480860 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2b2\" (UniqueName: \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.481230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.481207 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.481378 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.481355 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.519874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.519842 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:00.545462 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.545436 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfplf\" (UniqueName: \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\") pod \"maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.546237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.546211 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2b2\" (UniqueName: \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\") pod \"maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.575466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.575429 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:00.595013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.594977 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:00.602593 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.602170 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:00.624448 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.624419 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:00.912119 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:00.912086 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94e16ac_6b4f_4a0f_937d_d2af4e0c7489.slice/crio-5d348649bf85704b76fee82e2334e8a1eb1d78e213954c6861d92692689213ae WatchSource:0}: Error finding container 5d348649bf85704b76fee82e2334e8a1eb1d78e213954c6861d92692689213ae: Status 404 returned error can't find the container with id 5d348649bf85704b76fee82e2334e8a1eb1d78e213954c6861d92692689213ae Oct 31 00:45:00.915079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:00.915046 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:01.178163 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:01.178133 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1e8f37_2727_4964_a9a2_f087ae4a3199.slice/crio-7f83dcc714199fad2cf67fdf870bed3625318538951480731fd0672024b0ab0f WatchSource:0}: Error finding container 7f83dcc714199fad2cf67fdf870bed3625318538951480731fd0672024b0ab0f: Status 404 returned error can't find the container with id 7f83dcc714199fad2cf67fdf870bed3625318538951480731fd0672024b0ab0f Oct 31 00:45:01.181563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.181521 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:01.191488 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:01.191457 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40900a21_804a_4edb_9c37_471cc153de87.slice/crio-5f0b629c65ab6cfff072e22b47bad152d4353defa56184c6cc00c58e5812a4a1 WatchSource:0}: Error finding container 5f0b629c65ab6cfff072e22b47bad152d4353defa56184c6cc00c58e5812a4a1: Status 404 returned error can't find the container with id 5f0b629c65ab6cfff072e22b47bad152d4353defa56184c6cc00c58e5812a4a1 Oct 31 00:45:01.194616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.194448 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:01.226168 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:01.226141 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcc916d_dabb_43e6_85e6_1a6181422096.slice/crio-9501d1ccd73652e6e03cf0b83d659726a4edcdff8be7ddb93bd522996ab5058e WatchSource:0}: Error finding container 9501d1ccd73652e6e03cf0b83d659726a4edcdff8be7ddb93bd522996ab5058e: Status 404 returned error can't find the container with id 9501d1ccd73652e6e03cf0b83d659726a4edcdff8be7ddb93bd522996ab5058e Oct 31 00:45:01.228507 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.228484 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:01.229431 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:01.229407 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a30627_4d10_49d5_aed0_7591bfee4b30.slice/crio-8b3720205e1870f3ab8d79f2769cead621c160895369bc640aaa734036a2969e WatchSource:0}: Error finding container 8b3720205e1870f3ab8d79f2769cead621c160895369bc640aaa734036a2969e: Status 404 returned error can't find the container with id 8b3720205e1870f3ab8d79f2769cead621c160895369bc640aaa734036a2969e Oct 31 00:45:01.232326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.232305 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:01.657539 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.657500 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:01.875760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.875722 4471 generic.go:356] "Generic (PLEG): container finished" podID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerID="2e830cc6c1626c76822ad3464bbc85d7d7402d96653d8a54350d3d13a2dfaea7" exitCode=60 Oct 31 00:45:01.875956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.875765 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerDied","Data":"2e830cc6c1626c76822ad3464bbc85d7d7402d96653d8a54350d3d13a2dfaea7"} Oct 31 00:45:01.875956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.875825 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerStarted","Data":"5d348649bf85704b76fee82e2334e8a1eb1d78e213954c6861d92692689213ae"} Oct 31 00:45:01.876179 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.876161 4471 scope.go:122] "RemoveContainer" containerID="2e830cc6c1626c76822ad3464bbc85d7d7402d96653d8a54350d3d13a2dfaea7" Oct 31 00:45:01.878389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.878364 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerStarted","Data":"6a6bd5e5edbdd7d2617f3ef3add0691c4a0a5fd856a8c56880bc0b4c03af9a55"} Oct 31 00:45:01.878486 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.878414 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerStarted","Data":"9501d1ccd73652e6e03cf0b83d659726a4edcdff8be7ddb93bd522996ab5058e"} Oct 31 00:45:01.880020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.880001 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerStarted","Data":"9a28ffc84acece18a175827ec9a31634b127cfeaed8d5e7ea7c46d98ec984153"} Oct 31 00:45:01.880070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.880046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerStarted","Data":"5f0b629c65ab6cfff072e22b47bad152d4353defa56184c6cc00c58e5812a4a1"} Oct 31 00:45:01.881701 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.881680 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerStarted","Data":"9330dd2b8fe71a62616b4e4c0e7da097f9b82a6722738faff9453d12e62033b5"} Oct 31 00:45:01.881813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.881725 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerStarted","Data":"8b3720205e1870f3ab8d79f2769cead621c160895369bc640aaa734036a2969e"} Oct 31 00:45:01.883305 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.883282 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerStarted","Data":"d6813183fa288110f2fa1ee7b17bcca3b1dc2b79e956c750608f86657cac9cf2"} Oct 31 00:45:01.883371 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.883325 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerStarted","Data":"7f83dcc714199fad2cf67fdf870bed3625318538951480731fd0672024b0ab0f"} Oct 31 00:45:01.947750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:01.947680 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.157556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.157159 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" podStartSLOduration=2.157134834 podStartE2EDuration="2.157134834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.152484019 +0000 UTC m=+1829.077752973" watchObservedRunningTime="2026-10-31 00:45:02.157134834 +0000 UTC m=+1829.082403787" Oct 31 00:45:02.238703 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.238670 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.274114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.274065 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" podStartSLOduration=2.2740516680000002 podStartE2EDuration="2.274051668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.273611525 +0000 UTC m=+1829.198880475" watchObservedRunningTime="2026-10-31 00:45:02.274051668 +0000 UTC m=+1829.199320620" Oct 31 00:45:02.404972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.404935 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.406241 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.406200 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" podStartSLOduration=2.406185696 podStartE2EDuration="2.406185696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.402566838 +0000 UTC m=+1829.327835784" watchObservedRunningTime="2026-10-31 00:45:02.406185696 +0000 UTC m=+1829.331454648" Oct 31 00:45:02.528467 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.528345 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" podStartSLOduration=2.528330911 podStartE2EDuration="2.528330911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:02.526101428 +0000 UTC m=+1829.451370381" watchObservedRunningTime="2026-10-31 00:45:02.528330911 +0000 UTC m=+1829.453599864" Oct 31 00:45:02.893152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.893119 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerStarted","Data":"eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86"} Oct 31 00:45:02.894967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.894939 4471 generic.go:356] "Generic (PLEG): container finished" podID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerID="6a6bd5e5edbdd7d2617f3ef3add0691c4a0a5fd856a8c56880bc0b4c03af9a55" exitCode=60 Oct 31 00:45:02.895108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.895026 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerDied","Data":"6a6bd5e5edbdd7d2617f3ef3add0691c4a0a5fd856a8c56880bc0b4c03af9a55"} Oct 31 00:45:02.895287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.895272 4471 scope.go:122] "RemoveContainer" containerID="6a6bd5e5edbdd7d2617f3ef3add0691c4a0a5fd856a8c56880bc0b4c03af9a55" Oct 31 00:45:02.896933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.896898 4471 generic.go:356] "Generic (PLEG): container finished" podID="40900a21-804a-4edb-9c37-471cc153de87" containerID="9a28ffc84acece18a175827ec9a31634b127cfeaed8d5e7ea7c46d98ec984153" exitCode=60 Oct 31 00:45:02.897047 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.896984 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerDied","Data":"9a28ffc84acece18a175827ec9a31634b127cfeaed8d5e7ea7c46d98ec984153"} Oct 31 00:45:02.897214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.897196 4471 scope.go:122] "RemoveContainer" containerID="9a28ffc84acece18a175827ec9a31634b127cfeaed8d5e7ea7c46d98ec984153" Oct 31 00:45:02.898651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.898631 4471 generic.go:356] "Generic (PLEG): container finished" podID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerID="9330dd2b8fe71a62616b4e4c0e7da097f9b82a6722738faff9453d12e62033b5" exitCode=60 Oct 31 00:45:02.898863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.898845 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerDied","Data":"9330dd2b8fe71a62616b4e4c0e7da097f9b82a6722738faff9453d12e62033b5"} Oct 31 00:45:02.898979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.898965 4471 scope.go:122] "RemoveContainer" containerID="9330dd2b8fe71a62616b4e4c0e7da097f9b82a6722738faff9453d12e62033b5" Oct 31 00:45:02.902036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.902011 4471 generic.go:356] "Generic (PLEG): container finished" podID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerID="d6813183fa288110f2fa1ee7b17bcca3b1dc2b79e956c750608f86657cac9cf2" exitCode=60 Oct 31 00:45:02.902136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.902090 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerDied","Data":"d6813183fa288110f2fa1ee7b17bcca3b1dc2b79e956c750608f86657cac9cf2"} Oct 31 00:45:02.902301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:02.902284 4471 scope.go:122] "RemoveContainer" containerID="d6813183fa288110f2fa1ee7b17bcca3b1dc2b79e956c750608f86657cac9cf2" Oct 31 00:45:03.011423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.011373 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" podStartSLOduration=3.011359598 podStartE2EDuration="3.011359598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.01063082 +0000 UTC m=+1829.935899770" watchObservedRunningTime="2026-10-31 00:45:03.011359598 +0000 UTC m=+1829.936628550" Oct 31 00:45:03.909804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.909766 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerStarted","Data":"674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6"} Oct 31 00:45:03.911702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.911676 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerStarted","Data":"1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5"} Oct 31 00:45:03.913595 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.913566 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerStarted","Data":"d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b"} Oct 31 00:45:03.915385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.915361 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerStarted","Data":"feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8"} Oct 31 00:45:03.917158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.917135 4471 generic.go:356] "Generic (PLEG): container finished" podID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerID="eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86" exitCode=60 Oct 31 00:45:03.917485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.917176 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerDied","Data":"eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86"} Oct 31 00:45:03.917485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.917240 4471 scope.go:122] "RemoveContainer" containerID="2e830cc6c1626c76822ad3464bbc85d7d7402d96653d8a54350d3d13a2dfaea7" Oct 31 00:45:03.917630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:03.917502 4471 scope.go:122] "RemoveContainer" containerID="eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86" Oct 31 00:45:03.917987 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:03.917832 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx_odh-ai-gateway-infra(d94e16ac-6b4f-4a0f-937d-d2af4e0c7489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" Oct 31 00:45:04.281517 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.281484 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:45:04.924948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.924906 4471 generic.go:356] "Generic (PLEG): container finished" podID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerID="674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6" exitCode=60 Oct 31 00:45:04.925445 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.924994 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerDied","Data":"674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6"} Oct 31 00:45:04.925445 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.925106 4471 scope.go:122] "RemoveContainer" containerID="6a6bd5e5edbdd7d2617f3ef3add0691c4a0a5fd856a8c56880bc0b4c03af9a55" Oct 31 00:45:04.925445 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.925308 4471 scope.go:122] "RemoveContainer" containerID="674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6" Oct 31 00:45:04.925737 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.925712 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5_odh-ai-gateway-infra(1dcc916d-dabb-43e6-85e6-1a6181422096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" Oct 31 00:45:04.926901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.926877 4471 generic.go:356] "Generic (PLEG): container finished" podID="40900a21-804a-4edb-9c37-471cc153de87" containerID="1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5" exitCode=60 Oct 31 00:45:04.926982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.926948 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerDied","Data":"1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5"} Oct 31 00:45:04.927221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.927203 4471 scope.go:122] "RemoveContainer" containerID="1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5" Oct 31 00:45:04.927575 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.927515 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7_odh-ai-gateway-infra(40900a21-804a-4edb-9c37-471cc153de87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" podUID="40900a21-804a-4edb-9c37-471cc153de87" Oct 31 00:45:04.928625 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.928606 4471 generic.go:356] "Generic (PLEG): container finished" podID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerID="d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b" exitCode=60 Oct 31 00:45:04.928708 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.928684 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerDied","Data":"d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b"} Oct 31 00:45:04.928935 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.928921 4471 scope.go:122] "RemoveContainer" containerID="d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b" Oct 31 00:45:04.929228 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.929206 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4_odh-ai-gateway-infra(35a30627-4d10-49d5-aed0-7591bfee4b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" Oct 31 00:45:04.930461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.930442 4471 generic.go:356] "Generic (PLEG): container finished" podID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerID="feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8" exitCode=60 Oct 31 00:45:04.930577 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.930474 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerDied","Data":"feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8"} Oct 31 00:45:04.930831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.930811 4471 scope.go:122] "RemoveContainer" containerID="feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8" Oct 31 00:45:04.931177 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.931152 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4_odh-ai-gateway-infra(4b1e8f37-2727-4964-a9a2-f087ae4a3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" Oct 31 00:45:04.932419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.932403 4471 scope.go:122] "RemoveContainer" containerID="eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86" Oct 31 00:45:04.932650 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:04.932625 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx_odh-ai-gateway-infra(d94e16ac-6b4f-4a0f-937d-d2af4e0c7489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" Oct 31 00:45:04.939253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.939229 4471 scope.go:122] "RemoveContainer" containerID="9a28ffc84acece18a175827ec9a31634b127cfeaed8d5e7ea7c46d98ec984153" Oct 31 00:45:04.952204 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.952185 4471 scope.go:122] "RemoveContainer" containerID="9330dd2b8fe71a62616b4e4c0e7da097f9b82a6722738faff9453d12e62033b5" Oct 31 00:45:04.965090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:04.965072 4471 scope.go:122] "RemoveContainer" containerID="d6813183fa288110f2fa1ee7b17bcca3b1dc2b79e956c750608f86657cac9cf2" Oct 31 00:45:05.939730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:05.939704 4471 scope.go:122] "RemoveContainer" containerID="674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6" Oct 31 00:45:05.940204 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:05.940091 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5_odh-ai-gateway-infra(1dcc916d-dabb-43e6-85e6-1a6181422096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" Oct 31 00:45:05.941437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:05.941414 4471 scope.go:122] "RemoveContainer" containerID="1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5" Oct 31 00:45:05.941748 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:05.941726 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7_odh-ai-gateway-infra(40900a21-804a-4edb-9c37-471cc153de87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" podUID="40900a21-804a-4edb-9c37-471cc153de87" Oct 31 00:45:05.942917 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:05.942899 4471 scope.go:122] "RemoveContainer" containerID="d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b" Oct 31 00:45:05.943198 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:05.943174 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4_odh-ai-gateway-infra(35a30627-4d10-49d5-aed0-7591bfee4b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" Oct 31 00:45:05.944308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:05.944293 4471 scope.go:122] "RemoveContainer" containerID="feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8" Oct 31 00:45:05.944512 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:05.944497 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4_odh-ai-gateway-infra(4b1e8f37-2727-4964-a9a2-f087ae4a3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" Oct 31 00:45:06.683309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:06.683261 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:06.853075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:06.853044 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.025668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:07.025585 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:07.190553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:07.187951 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:10.906876 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:10.906840 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:12.514507 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:12.514456 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:15.202489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:15.202414 4471 scope.go:122] "RemoveContainer" containerID="eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86" Oct 31 00:45:16.033085 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:16.033043 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerStarted","Data":"9a40d3e4b29b4398df858e4f11847d335c405460d5e199c559e480adf6de0ddd"} Oct 31 00:45:16.203235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:16.203197 4471 scope.go:122] "RemoveContainer" containerID="674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6" Oct 31 00:45:17.044523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.044488 4471 generic.go:356] "Generic (PLEG): container finished" podID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerID="9a40d3e4b29b4398df858e4f11847d335c405460d5e199c559e480adf6de0ddd" exitCode=60 Oct 31 00:45:17.044698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.044573 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerDied","Data":"9a40d3e4b29b4398df858e4f11847d335c405460d5e199c559e480adf6de0ddd"} Oct 31 00:45:17.044698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.044667 4471 scope.go:122] "RemoveContainer" containerID="eeb52337862239af53eb0112b3039c49f96f64f0e0003ed996f4743a213e9f86" Oct 31 00:45:17.045130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.045106 4471 scope.go:122] "RemoveContainer" containerID="9a40d3e4b29b4398df858e4f11847d335c405460d5e199c559e480adf6de0ddd" Oct 31 00:45:17.045870 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:17.045842 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx_odh-ai-gateway-infra(d94e16ac-6b4f-4a0f-937d-d2af4e0c7489)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" Oct 31 00:45:17.047235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.047212 4471 generic.go:356] "Generic (PLEG): container finished" podID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerID="5cd5081b528794b62d8dd6d68ced15efab13f418a734979098b0066f1f6918ff" exitCode=60 Oct 31 00:45:17.047318 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.047243 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerDied","Data":"5cd5081b528794b62d8dd6d68ced15efab13f418a734979098b0066f1f6918ff"} Oct 31 00:45:17.047686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.047669 4471 scope.go:122] "RemoveContainer" containerID="5cd5081b528794b62d8dd6d68ced15efab13f418a734979098b0066f1f6918ff" Oct 31 00:45:17.047986 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:17.047968 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5_odh-ai-gateway-infra(1dcc916d-dabb-43e6-85e6-1a6181422096)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" Oct 31 00:45:17.058019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.058000 4471 scope.go:122] "RemoveContainer" containerID="674d1acac44b1ce2645f1002378f047bea9af22d487874f2cc41c3827ca634f6" Oct 31 00:45:17.161567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.161509 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:17.202552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.202497 4471 scope.go:122] "RemoveContainer" containerID="feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8" Oct 31 00:45:17.202748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.202617 4471 scope.go:122] "RemoveContainer" containerID="1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5" Oct 31 00:45:17.202748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:17.202699 4471 scope.go:122] "RemoveContainer" containerID="d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b" Oct 31 00:45:18.055210 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.055170 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerStarted","Data":"6266e1b250ae608dcc654635637b3b1800023e395c54d85b10128abb6b5e1d9e"} Oct 31 00:45:18.058472 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.058438 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerStarted","Data":"c1611c97c77ae09c27be564e2116739b04c345fdce1465a20681e036884ad1ee"} Oct 31 00:45:18.064993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.064965 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerStarted","Data":"14bacfff3572fc54591f9433d2f7430edd08c894cd4df973ea6346f8f9594e71"} Oct 31 00:45:18.084086 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.084052 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:18.248495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.248461 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:18.309698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.309624 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:18.405701 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.405627 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\") pod \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " Oct 31 00:45:18.407572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.406028 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\" (UniqueName: \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\") pod \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\" (UID: \"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489\") " Oct 31 00:45:18.407572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.406179 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca" pod "d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" (UID: "d94e16ac-6b4f-4a0f-937d-d2af4e0c7489"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.407572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.407136 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.412112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.412024 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5" pod "d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" (UID: "d94e16ac-6b4f-4a0f-937d-d2af4e0c7489"). InnerVolumeSpecName "kube-api-access-xsdv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.508338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.508300 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsdv5\" (UniqueName: \"kubernetes.io/projected/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489-kube-api-access-xsdv5\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.544522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.544489 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:18.609083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.609003 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\") pod \"1dcc916d-dabb-43e6-85e6-1a6181422096\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " Oct 31 00:45:18.609220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.609173 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\" (UniqueName: \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\") pod \"1dcc916d-dabb-43e6-85e6-1a6181422096\" (UID: \"1dcc916d-dabb-43e6-85e6-1a6181422096\") " Oct 31 00:45:18.609420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.609394 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca" pod "1dcc916d-dabb-43e6-85e6-1a6181422096" (UID: "1dcc916d-dabb-43e6-85e6-1a6181422096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:18.609617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.609603 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc916d-dabb-43e6-85e6-1a6181422096-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.611227 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.611208 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q" pod "1dcc916d-dabb-43e6-85e6-1a6181422096" (UID: "1dcc916d-dabb-43e6-85e6-1a6181422096"). InnerVolumeSpecName "kube-api-access-fzx7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.711047 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:18.711018 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzx7q\" (UniqueName: \"kubernetes.io/projected/1dcc916d-dabb-43e6-85e6-1a6181422096-kube-api-access-fzx7q\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.089003 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.088968 4471 generic.go:356] "Generic (PLEG): container finished" podID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerID="c1611c97c77ae09c27be564e2116739b04c345fdce1465a20681e036884ad1ee" exitCode=60 Oct 31 00:45:19.089481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.089040 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerDied","Data":"c1611c97c77ae09c27be564e2116739b04c345fdce1465a20681e036884ad1ee"} Oct 31 00:45:19.089481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.089100 4471 scope.go:122] "RemoveContainer" containerID="feeacfaea15a8935632e7cb9be9c64718f66a2e5bee769fad6de791f4a47b7d8" Oct 31 00:45:19.089611 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.089489 4471 scope.go:122] "RemoveContainer" containerID="c1611c97c77ae09c27be564e2116739b04c345fdce1465a20681e036884ad1ee" Oct 31 00:45:19.089938 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:19.089910 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4_odh-ai-gateway-infra(4b1e8f37-2727-4964-a9a2-f087ae4a3199)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" Oct 31 00:45:19.090628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.090608 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" Oct 31 00:45:19.090719 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.090636 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx" event={"ID":"d94e16ac-6b4f-4a0f-937d-d2af4e0c7489","Type":"ContainerDied","Data":"5d348649bf85704b76fee82e2334e8a1eb1d78e213954c6861d92692689213ae"} Oct 31 00:45:19.092041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.092014 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" Oct 31 00:45:19.092135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.092061 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5" event={"ID":"1dcc916d-dabb-43e6-85e6-1a6181422096","Type":"ContainerDied","Data":"9501d1ccd73652e6e03cf0b83d659726a4edcdff8be7ddb93bd522996ab5058e"} Oct 31 00:45:19.096109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.095196 4471 generic.go:356] "Generic (PLEG): container finished" podID="40900a21-804a-4edb-9c37-471cc153de87" containerID="14bacfff3572fc54591f9433d2f7430edd08c894cd4df973ea6346f8f9594e71" exitCode=60 Oct 31 00:45:19.096109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.095282 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerDied","Data":"14bacfff3572fc54591f9433d2f7430edd08c894cd4df973ea6346f8f9594e71"} Oct 31 00:45:19.096109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.095744 4471 scope.go:122] "RemoveContainer" containerID="14bacfff3572fc54591f9433d2f7430edd08c894cd4df973ea6346f8f9594e71" Oct 31 00:45:19.096109 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:19.096062 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7_odh-ai-gateway-infra(40900a21-804a-4edb-9c37-471cc153de87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" podUID="40900a21-804a-4edb-9c37-471cc153de87" Oct 31 00:45:19.097647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.097627 4471 generic.go:356] "Generic (PLEG): container finished" podID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerID="6266e1b250ae608dcc654635637b3b1800023e395c54d85b10128abb6b5e1d9e" exitCode=60 Oct 31 00:45:19.097752 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.097680 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerDied","Data":"6266e1b250ae608dcc654635637b3b1800023e395c54d85b10128abb6b5e1d9e"} Oct 31 00:45:19.098045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.098029 4471 scope.go:122] "RemoveContainer" containerID="6266e1b250ae608dcc654635637b3b1800023e395c54d85b10128abb6b5e1d9e" Oct 31 00:45:19.098286 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:19.098266 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4_odh-ai-gateway-infra(35a30627-4d10-49d5-aed0-7591bfee4b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" Oct 31 00:45:19.103995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.103976 4471 scope.go:122] "RemoveContainer" containerID="9a40d3e4b29b4398df858e4f11847d335c405460d5e199c559e480adf6de0ddd" Oct 31 00:45:19.114075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.114054 4471 scope.go:122] "RemoveContainer" containerID="5cd5081b528794b62d8dd6d68ced15efab13f418a734979098b0066f1f6918ff" Oct 31 00:45:19.124138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.124104 4471 scope.go:122] "RemoveContainer" containerID="1bee93dce01a31decc60ace4656acfb04b5571197c7d8e479cf1c92e36be5af5" Oct 31 00:45:19.137623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.137507 4471 scope.go:122] "RemoveContainer" containerID="d8d7b91de5fee9ba1dd869a206dacf56d64f841fc387cd095c232c7ed2e2068b" Oct 31 00:45:19.188048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.188007 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:19.304212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.304175 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:19.421752 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.421720 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:19.484332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.484287 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:19.487450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.487423 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4b86e4-29890125-9lqxx"] Oct 31 00:45:19.704524 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.704443 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:19.714246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:19.714215 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f32ec2-29890125-xzxc5"] Oct 31 00:45:20.400846 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.400783 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:20.446957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.446927 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\") pod \"40900a21-804a-4edb-9c37-471cc153de87\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " Oct 31 00:45:20.447142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.447127 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\" (UniqueName: \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\") pod \"40900a21-804a-4edb-9c37-471cc153de87\" (UID: \"40900a21-804a-4edb-9c37-471cc153de87\") " Oct 31 00:45:20.448306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.448282 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca" pod "40900a21-804a-4edb-9c37-471cc153de87" (UID: "40900a21-804a-4edb-9c37-471cc153de87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.451271 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.451244 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2" pod "40900a21-804a-4edb-9c37-471cc153de87" (UID: "40900a21-804a-4edb-9c37-471cc153de87"). InnerVolumeSpecName "kube-api-access-ss2b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.526057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.526025 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:20.527996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.527973 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:20.549683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.549644 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2b2\" (UniqueName: \"kubernetes.io/projected/40900a21-804a-4edb-9c37-471cc153de87-kube-api-access-ss2b2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.549683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.549683 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40900a21-804a-4edb-9c37-471cc153de87-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.650924 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.650836 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\") pod \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " Oct 31 00:45:20.651067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.650950 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\") pod \"35a30627-4d10-49d5-aed0-7591bfee4b30\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " Oct 31 00:45:20.651149 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651130 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\" (UniqueName: \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\") pod \"35a30627-4d10-49d5-aed0-7591bfee4b30\" (UID: \"35a30627-4d10-49d5-aed0-7591bfee4b30\") " Oct 31 00:45:20.651242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651226 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\" (UniqueName: \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\") pod \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\" (UID: \"4b1e8f37-2727-4964-a9a2-f087ae4a3199\") " Oct 31 00:45:20.651242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651233 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca" pod "4b1e8f37-2727-4964-a9a2-f087ae4a3199" (UID: "4b1e8f37-2727-4964-a9a2-f087ae4a3199"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.651565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651545 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca" pod "35a30627-4d10-49d5-aed0-7591bfee4b30" (UID: "35a30627-4d10-49d5-aed0-7591bfee4b30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.651939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651916 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e8f37-2727-4964-a9a2-f087ae4a3199-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.652069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.651944 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a30627-4d10-49d5-aed0-7591bfee4b30-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.653711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.653689 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww" pod "4b1e8f37-2727-4964-a9a2-f087ae4a3199" (UID: "4b1e8f37-2727-4964-a9a2-f087ae4a3199"). InnerVolumeSpecName "kube-api-access-sgcww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.655460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.655437 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf" pod "35a30627-4d10-49d5-aed0-7591bfee4b30" (UID: "35a30627-4d10-49d5-aed0-7591bfee4b30"). InnerVolumeSpecName "kube-api-access-vfplf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.755833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.755796 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfplf\" (UniqueName: \"kubernetes.io/projected/35a30627-4d10-49d5-aed0-7591bfee4b30-kube-api-access-vfplf\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.756057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:20.756029 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgcww\" (UniqueName: \"kubernetes.io/projected/4b1e8f37-2727-4964-a9a2-f087ae4a3199-kube-api-access-sgcww\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:21.126573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.126467 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" event={"ID":"40900a21-804a-4edb-9c37-471cc153de87","Type":"ContainerDied","Data":"5f0b629c65ab6cfff072e22b47bad152d4353defa56184c6cc00c58e5812a4a1"} Oct 31 00:45:21.126573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.126473 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7" Oct 31 00:45:21.126573 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.126556 4471 scope.go:122] "RemoveContainer" containerID="14bacfff3572fc54591f9433d2f7430edd08c894cd4df973ea6346f8f9594e71" Oct 31 00:45:21.128114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.128100 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" Oct 31 00:45:21.128223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.128139 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4" event={"ID":"35a30627-4d10-49d5-aed0-7591bfee4b30","Type":"ContainerDied","Data":"8b3720205e1870f3ab8d79f2769cead621c160895369bc640aaa734036a2969e"} Oct 31 00:45:21.130507 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.130480 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" Oct 31 00:45:21.130628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.130514 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4" event={"ID":"4b1e8f37-2727-4964-a9a2-f087ae4a3199","Type":"ContainerDied","Data":"7f83dcc714199fad2cf67fdf870bed3625318538951480731fd0672024b0ab0f"} Oct 31 00:45:21.139474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.139457 4471 scope.go:122] "RemoveContainer" containerID="6266e1b250ae608dcc654635637b3b1800023e395c54d85b10128abb6b5e1d9e" Oct 31 00:45:21.150277 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.150260 4471 scope.go:122] "RemoveContainer" containerID="c1611c97c77ae09c27be564e2116739b04c345fdce1465a20681e036884ad1ee" Oct 31 00:45:21.210908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.210872 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" path="/var/lib/kubelet/pods/1dcc916d-dabb-43e6-85e6-1a6181422096/volumes" Oct 31 00:45:21.211311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.211298 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" path="/var/lib/kubelet/pods/d94e16ac-6b4f-4a0f-937d-d2af4e0c7489/volumes" Oct 31 00:45:21.298811 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.298514 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:21.349406 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.349372 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6f4a5-29890125-2lcn4"] Oct 31 00:45:21.467008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.466977 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:21.475091 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.475047 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e8ba06-29890125-26cv4"] Oct 31 00:45:21.587153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.587115 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:21.590470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:21.590445 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-412743-29890125-b7cl7"] Oct 31 00:45:23.212291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:23.212254 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" path="/var/lib/kubelet/pods/35a30627-4d10-49d5-aed0-7591bfee4b30/volumes" Oct 31 00:45:23.212799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:23.212783 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40900a21-804a-4edb-9c37-471cc153de87" path="/var/lib/kubelet/pods/40900a21-804a-4edb-9c37-471cc153de87/volumes" Oct 31 00:45:23.213167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:23.213152 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" path="/var/lib/kubelet/pods/4b1e8f37-2727-4964-a9a2-f087ae4a3199/volumes" Oct 31 00:45:23.409869 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:23.409828 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:26.274066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:26.274030 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:27.173103 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.173068 4471 generic.go:356] "Generic (PLEG): container finished" podID="cbe7a287-b617-42f4-b531-a891fe920719" containerID="f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89" exitCode=0 Oct 31 00:45:27.173306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.173156 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerDied","Data":"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89"} Oct 31 00:45:27.173639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.173624 4471 scope.go:122] "RemoveContainer" containerID="f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89" Oct 31 00:45:27.175168 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.175141 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:27.175226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.175212 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:27.175259 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:27.175243 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:28.192487 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:28.192441 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerStarted","Data":"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286"} Oct 31 00:45:28.491157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:28.491075 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:28.836110 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:28.836018 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:45:28.836266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:28.836199 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:45:28.836266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:28.836248 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:45:29.836458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:29.836425 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:45:29.836893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:29.836479 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:45:30.836738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:30.836699 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:45:30.837169 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:30.836756 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:45:31.835739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:31.835705 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:45:31.835939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:31.835770 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:45:32.835860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:32.835824 4471 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:45:32.836263 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:32.835879 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 00:45:33.468728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:33.468686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:33.840332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:33.840242 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:45:33.840814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:33.840651 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:45:33.841369 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:33.841351 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:45:34.283246 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:34.283209 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:45:38.356872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:38.356838 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:43.385915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:43.385879 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:46.759252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:46.759215 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:45:46.759739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:46.759423 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" containerID="cri-o://524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08" gracePeriod=30 Oct 31 00:45:47.150412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:47.150377 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:45:47.150664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:47.150638 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" containerID="cri-o://7f833c10e0fd36c7620bdc4bb3d12461ba65e70704e6a50001f4d90dae66fd68" gracePeriod=30 Oct 31 00:45:48.212400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:48.212364 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.377918 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:48.377881 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:48.526284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:48.526196 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.694883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:50.694848 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:50.838400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:50.838350 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:51.530922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:51.530874 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:51.994915 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:51.994883 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.037821 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.037761 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:45:52.118485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118447 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118507 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118574 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118601 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118642 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118762 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118780 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.118970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118846 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.118970 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118906 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.119134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118971 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.119134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119037 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\") pod \"ac398aec-72bf-4698-a1dd-83133739184b\" (UID: \"ac398aec-72bf-4698-a1dd-83133739184b\") " Oct 31 00:45:52.119134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.118969 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.119294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119139 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.119294 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119171 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:52.119629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119600 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.119738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119631 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.119738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119645 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.119738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119658 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac398aec-72bf-4698-a1dd-83133739184b-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.119738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.119673 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.121008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.120981 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.121099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.120979 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:45:52.121099 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.121013 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "kube-api-access-2d8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.121170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.121139 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy" pod "ac398aec-72bf-4698-a1dd-83133739184b" (UID: "ac398aec-72bf-4698-a1dd-83133739184b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.220681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.220628 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8cn\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-kube-api-access-2d8cn\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.220681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.220662 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac398aec-72bf-4698-a1dd-83133739184b-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.220681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.220677 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac398aec-72bf-4698-a1dd-83133739184b-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.220903 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.220690 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac398aec-72bf-4698-a1dd-83133739184b-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.361733 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.361696 4471 generic.go:356] "Generic (PLEG): container finished" podID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerID="7f833c10e0fd36c7620bdc4bb3d12461ba65e70704e6a50001f4d90dae66fd68" exitCode=0 Oct 31 00:45:52.361911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.361818 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" event={"ID":"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b","Type":"ContainerDied","Data":"7f833c10e0fd36c7620bdc4bb3d12461ba65e70704e6a50001f4d90dae66fd68"} Oct 31 00:45:52.367214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.367177 4471 generic.go:356] "Generic (PLEG): container finished" podID="ac398aec-72bf-4698-a1dd-83133739184b" containerID="524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08" exitCode=0 Oct 31 00:45:52.367399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.367382 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" event={"ID":"ac398aec-72bf-4698-a1dd-83133739184b","Type":"ContainerDied","Data":"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08"} Oct 31 00:45:52.367476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.367434 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" event={"ID":"ac398aec-72bf-4698-a1dd-83133739184b","Type":"ContainerDied","Data":"0ae191189fb6e1af5957301a5375001229ed6051c6caf96cfce2c448d51c9b36"} Oct 31 00:45:52.367476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.367463 4471 scope.go:122] "RemoveContainer" containerID="524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08" Oct 31 00:45:52.367668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.367655 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf" Oct 31 00:45:52.397695 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.397666 4471 scope.go:122] "RemoveContainer" containerID="524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08" Oct 31 00:45:52.398105 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:45:52.398077 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08\": container with ID starting with 524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08 not found: ID does not exist" containerID="524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08" Oct 31 00:45:52.398244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.398118 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08"} err="failed to get container status \"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08\": rpc error: code = NotFound desc = could not find container \"524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08\": container with ID starting with 524bc2b91cbf81cce6d4c28ac7cb2ebc0cc6656ac890fc8b00d4f5b861e7bc08 not found: ID does not exist" Oct 31 00:45:52.439175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.437916 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:45:52.525408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525358 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525484 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525588 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525674 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525684 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.525762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525750 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525783 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525811 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.525914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525818 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.526066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525912 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.526066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.525942 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\") pod \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\" (UID: \"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b\") " Oct 31 00:45:52.526167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526109 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.526167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526126 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:52.526646 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.526791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526758 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.526958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526933 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.526958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526960 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.527151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.526976 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.528313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.528288 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:45:52.528392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.528332 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "kube-api-access-c4l7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.528392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.528358 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:45:52.528521 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.528508 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token" pod "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" (UID: "d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:52.546401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.546371 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:45:52.550446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.550419 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-qm9sf"] Oct 31 00:45:52.628729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.628556 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4l7d\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-kube-api-access-c4l7d\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.628729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.628600 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.628729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.628616 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.628729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.628630 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.628729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.628644 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:52.798812 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.798779 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:45:52.799512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799489 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799511 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799539 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799548 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799558 4471 cpu_manager.go:414] "containerMap: removing container" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799565 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799575 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799581 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799596 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799601 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799609 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799619 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799630 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799635 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799645 4471 cpu_manager.go:414] "containerMap: removing container" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799650 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799660 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799664 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799672 4471 cpu_manager.go:414] "containerMap: removing container" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799676 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799686 4471 cpu_manager.go:414] "containerMap: removing container" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799690 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.799705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799696 4471 cpu_manager.go:414] "containerMap: removing container" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799700 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799727 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799732 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799739 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799744 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799755 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799760 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799902 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799910 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" containerName="istio-proxy" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799916 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799924 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799932 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799941 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799948 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799957 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799965 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799973 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799980 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799988 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.799995 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a30627-4d10-49d5-aed0-7591bfee4b30" containerName="cleanup" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.800005 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac398aec-72bf-4698-a1dd-83133739184b" containerName="istio-proxy" Oct 31 00:45:52.800952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.800011 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:45:52.803769 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.803753 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832296 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832265 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832390 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832503 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832473 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832561 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832541 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832645 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832789 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832775 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832802 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832896 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.832989 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.832972 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qxh\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.858200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.858166 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e9c3c1-openshift-default-dockercfg-782hg\"" Oct 31 00:45:52.862070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.862006 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:45:52.889597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.889460 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.934285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934244 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qxh\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934316 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934364 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934419 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934487 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934563 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934663 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934688 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.934833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.934738 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.935201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.935178 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.935763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.935741 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.936750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.936718 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.936884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.936857 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.936955 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.936910 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.938586 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.938563 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.938900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.938879 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.999225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.999188 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qxh\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:52.999632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:52.999432 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\") pod \"e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:53.047571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.045463 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:53.117275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.117233 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:53.215358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.212214 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac398aec-72bf-4698-a1dd-83133739184b" path="/var/lib/kubelet/pods/ac398aec-72bf-4698-a1dd-83133739184b/volumes" Oct 31 00:45:53.377024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.376991 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" Oct 31 00:45:53.377208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.377015 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc" event={"ID":"d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b","Type":"ContainerDied","Data":"4283da88de66eeab3bc797fac0e649ee1b6a875f040737237ffaad92de0c747f"} Oct 31 00:45:53.377208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.377091 4471 scope.go:122] "RemoveContainer" containerID="7f833c10e0fd36c7620bdc4bb3d12461ba65e70704e6a50001f4d90dae66fd68" Oct 31 00:45:53.539510 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:45:53.539478 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebca05b_284b_4c74_a31a_ac5b7c271e2a.slice/crio-9a684331ffac53dc24c90ea89ed754ff9ac1a83421b402d848baab3862e4148c WatchSource:0}: Error finding container 9a684331ffac53dc24c90ea89ed754ff9ac1a83421b402d848baab3862e4148c: Status 404 returned error can't find the container with id 9a684331ffac53dc24c90ea89ed754ff9ac1a83421b402d848baab3862e4148c Oct 31 00:45:53.543138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.543095 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:45:53.543282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.543124 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:53.543282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.543207 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:53.543282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.543250 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:45:53.547326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.547306 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:45:53.550581 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.550559 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-412743-openshift-default-7bddc8b5df-s66zc"] Oct 31 00:45:53.829892 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:53.829800 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:54.386895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:54.386861 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerStarted","Data":"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46"} Oct 31 00:45:54.387303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:54.386911 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerStarted","Data":"9a684331ffac53dc24c90ea89ed754ff9ac1a83421b402d848baab3862e4148c"} Oct 31 00:45:54.510449 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:54.510394 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podStartSLOduration=2.510380568 podStartE2EDuration="2.510380568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:54.508792514 +0000 UTC m=+1881.434061466" watchObservedRunningTime="2026-10-31 00:45:54.510380568 +0000 UTC m=+1881.435649520" Oct 31 00:45:55.118233 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:55.118197 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:55.124414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:55.124385 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:55.209667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:55.209625 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b" path="/var/lib/kubelet/pods/d64cc4b4-f2a4-4639-8b58-3b0d7e5e467b/volumes" Oct 31 00:45:55.393140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:55.393109 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:55.394078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:55.394057 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:45:57.372440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:57.372402 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:57.545157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:57.545119 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:57.753964 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:57.753931 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:59.375505 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:45:59.375466 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:03.695592 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:03.695511 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.281470 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:46:04.281429 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:46:04.588262 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:04.588172 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.768934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:04.768884 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:04.914474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:04.914433 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:05.169199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.169112 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:46:05.170302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.170275 4471 cpu_manager.go:414] "containerMap: removing container" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:46:05.170456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.170441 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40900a21-804a-4edb-9c37-471cc153de87" containerName="cleanup" Oct 31 00:46:05.170597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.170585 4471 cpu_manager.go:414] "containerMap: removing container" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:46:05.170680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.170669 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcc916d-dabb-43e6-85e6-1a6181422096" containerName="cleanup" Oct 31 00:46:05.170950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.170937 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d94e16ac-6b4f-4a0f-937d-d2af4e0c7489" containerName="cleanup" Oct 31 00:46:05.171041 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.171031 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1e8f37-2727-4964-a9a2-f087ae4a3199" containerName="cleanup" Oct 31 00:46:05.194211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.183021 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.236651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.236506 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:46:05.254330 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.254287 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27csn\" (UniqueName: \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.254520 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.254429 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.273571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.272164 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e9c3c1\"" Oct 31 00:46:05.273571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.272299 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e9c3c1-dockercfg-nlst6\"" Oct 31 00:46:05.354734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.354698 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:05.355689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.355661 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27csn\" (UniqueName: \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.355768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.355752 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.356460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.356440 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.417943 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.417910 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27csn\" (UniqueName: \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\") pod \"payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.530191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.530098 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:05.903820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.903786 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:46:05.908114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.908095 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.908670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.908648 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:46:05.966067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.966035 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.966404 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.966357 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxt7\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.966630 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.966609 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.966840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.966824 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.966967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.966953 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.967084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.967070 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.967300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.967261 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.967463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.967446 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:05.967678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:05.967662 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087597 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087700 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxt7\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087748 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087866 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087897 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.087960 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.088058 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.088108 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.088191 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.111166 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c0f56c-openshift-default-dockercfg-ng58l\"" Oct 31 00:46:06.127834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.115845 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:46:06.128667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.128587 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:06.140568 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.136899 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.149713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.147326 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.230461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.151816 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.230461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.175335 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.230461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.184190 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.233929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.233897 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.234344 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.234304 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxt7\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.236787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.236609 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.238243 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.237769 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\") pod \"e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.433565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.432285 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:06.471941 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.471908 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" event={"ID":"a9d97a53-f3b2-46d1-b89b-86925338f95d","Type":"ContainerStarted","Data":"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068"} Oct 31 00:46:06.472114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.471967 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" event={"ID":"a9d97a53-f3b2-46d1-b89b-86925338f95d","Type":"ContainerStarted","Data":"f39375d68e43c4d1641e55104c0941cf49ba1566fe7e23cb38a00b6c132881d8"} Oct 31 00:46:06.472114 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.471994 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:06.525773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.525685 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:06.698520 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.698465 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" podStartSLOduration=1.6984495769999999 podStartE2EDuration="1.698449577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:06.697578453 +0000 UTC m=+1893.622847437" watchObservedRunningTime="2026-10-31 00:46:06.698449577 +0000 UTC m=+1893.623718529" Oct 31 00:46:06.947644 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:46:06.947605 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413e0848_de9b_428e_8c32_5cbd16504047.slice/crio-9d82675d448e0cfb55bc2c72189d2f65cdada80b584be49ed39cd81deceddc05 WatchSource:0}: Error finding container 9d82675d448e0cfb55bc2c72189d2f65cdada80b584be49ed39cd81deceddc05: Status 404 returned error can't find the container with id 9d82675d448e0cfb55bc2c72189d2f65cdada80b584be49ed39cd81deceddc05 Oct 31 00:46:06.950308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.950284 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:46:06.950705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.950674 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:06.950758 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.950742 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:06.950793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:06.950772 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:46:07.479888 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:07.479845 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerStarted","Data":"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445"} Oct 31 00:46:07.480088 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:07.479927 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerStarted","Data":"9d82675d448e0cfb55bc2c72189d2f65cdada80b584be49ed39cd81deceddc05"} Oct 31 00:46:07.526076 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:07.526046 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:07.602816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:07.602767 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" podStartSLOduration=2.602754159 podStartE2EDuration="2.602754159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:07.601543676 +0000 UTC m=+1894.526812626" watchObservedRunningTime="2026-10-31 00:46:07.602754159 +0000 UTC m=+1894.528023112" Oct 31 00:46:08.533144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:08.533114 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:09.492465 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:09.492437 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:09.493320 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:09.493298 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:46:10.309614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:10.309576 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.478852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:10.478823 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.695940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:10.695905 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.849768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:10.849739 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:10.999826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:10.999720 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:11.442979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:11.442944 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:12.343747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:12.343705 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:12.720854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:12.720813 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:13.704890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:13.704857 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:13.891922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:13.891049 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.664291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:15.664245 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:15.863816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:15.863777 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:16.016978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:16.016877 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.397899 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:17.397817 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.483884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:17.483852 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:46:17.561497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:17.561441 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.823940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:17.823830 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:20.629658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:20.629619 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:20.819219 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:20.819000 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.048457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.048421 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:46:21.052999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.052977 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.135563 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.135499 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.135957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.135934 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595bc\" (UniqueName: \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.159893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.159858 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2594a9\"" Oct 31 00:46:21.160290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.160270 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2594a9-dockercfg-npxsv\"" Oct 31 00:46:21.173158 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.173110 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:46:21.234246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.234208 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:46:21.242174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.241505 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.242921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.242892 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595bc\" (UniqueName: \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.243360 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.242541 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.243575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.241691 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.261440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.261408 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:46:21.306666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.306580 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-595bc\" (UniqueName: \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\") pod \"payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.311224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.311191 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:21.351288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.351238 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.351484 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.351367 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6qb\" (UniqueName: \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.369044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.369013 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:21.454178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.453241 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6qb\" (UniqueName: \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.454178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.453447 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.454978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.454863 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.517087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.517050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6qb\" (UniqueName: \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\") pod \"payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.575284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.575205 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:21.747638 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:46:21.747599 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c89e7e_e2bb_423a_b9fb_8c67a6219191.slice/crio-a61f3919418322c982f47058022147db0ebc4aff1ba73f387bd5febe959598d6 WatchSource:0}: Error finding container a61f3919418322c982f47058022147db0ebc4aff1ba73f387bd5febe959598d6: Status 404 returned error can't find the container with id a61f3919418322c982f47058022147db0ebc4aff1ba73f387bd5febe959598d6 Oct 31 00:46:21.750990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.750965 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:46:21.946907 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:46:21.946878 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594859f6_ff0c_43c0_87b3_3c46e2ec37d1.slice/crio-22de08f5d2d13deec1750a726dfb47708fa1ccf2b67bd85a19cedec01cb615d5 WatchSource:0}: Error finding container 22de08f5d2d13deec1750a726dfb47708fa1ccf2b67bd85a19cedec01cb615d5: Status 404 returned error can't find the container with id 22de08f5d2d13deec1750a726dfb47708fa1ccf2b67bd85a19cedec01cb615d5 Oct 31 00:46:21.950971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:21.950946 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:46:22.572945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.572906 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" event={"ID":"594859f6-ff0c-43c0-87b3-3c46e2ec37d1","Type":"ContainerStarted","Data":"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a"} Oct 31 00:46:22.573145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.572969 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" event={"ID":"594859f6-ff0c-43c0-87b3-3c46e2ec37d1","Type":"ContainerStarted","Data":"22de08f5d2d13deec1750a726dfb47708fa1ccf2b67bd85a19cedec01cb615d5"} Oct 31 00:46:22.573145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.573002 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:22.574706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.574679 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" event={"ID":"94c89e7e-e2bb-423a-b9fb-8c67a6219191","Type":"ContainerStarted","Data":"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d"} Oct 31 00:46:22.574854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.574722 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" event={"ID":"94c89e7e-e2bb-423a-b9fb-8c67a6219191","Type":"ContainerStarted","Data":"a61f3919418322c982f47058022147db0ebc4aff1ba73f387bd5febe959598d6"} Oct 31 00:46:22.574854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.574743 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:22.693457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.693403 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" podStartSLOduration=1.6933898200000002 podStartE2EDuration="1.69338982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:22.690310501 +0000 UTC m=+1909.615579455" watchObservedRunningTime="2026-10-31 00:46:22.69338982 +0000 UTC m=+1909.618658772" Oct 31 00:46:22.823394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:22.823253 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" podStartSLOduration=1.8232333299999999 podStartE2EDuration="1.82323333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:22.819724179 +0000 UTC m=+1909.744993141" watchObservedRunningTime="2026-10-31 00:46:22.82323333 +0000 UTC m=+1909.748502283" Oct 31 00:46:26.125164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:26.125129 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.291468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:26.291434 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.437193 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:26.437143 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:26.658667 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:26.658627 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:31.290357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:31.290322 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:33.583195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:33.583164 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:46:33.583601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:33.583243 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:46:34.281038 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:46:34.280998 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:46:36.291928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:36.291895 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:41.265077 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:41.265039 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:46.365515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:46.365413 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:51.376990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:51.376918 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:56.031996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:56.031958 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:56.176818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:56.176773 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:56.734772 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:46:56.734738 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:02.122143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:02.122103 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.956976 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:03.956937 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:47:04.070192 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:04.070156 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:47:04.282931 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:04.282843 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:47:04.298086 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:04.298049 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:04.499557 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:04.499512 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:07.406169 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:07.406135 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:08.957550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:08.957493 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" containerID="cri-o://467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b" gracePeriod=25 Oct 31 00:47:09.071613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:09.071568 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" containerID="cri-o://9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc" gracePeriod=25 Oct 31 00:47:12.557895 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:12.557857 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:47:12.712339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:12.712302 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:13.610359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:13.610312 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:47:18.185607 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:18.185561 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:20.955095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:20.955053 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:22.557886 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:22.557844 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:47:23.568709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:23.568672 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:23.616059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:23.615998 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:47:25.930393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:25.930356 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:26.765617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:26.765582 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:26.972494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:26.972455 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.250030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:27.249994 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.582877 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:27.582763 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:27.878940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:27.878849 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:28.724648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.724608 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:47:28.731730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.731697 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743550 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743652 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743693 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743759 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743869 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743857 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.743967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743953 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.744017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.743975 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck4jv\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.744105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.744079 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.798253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.798217 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:47:28.845926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.845890 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5b4db8-openshift-default-dockercfg-tc2fm\"" Oct 31 00:47:28.847640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.847483 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.848582 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.848539 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.848770 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.848755 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.849156 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.849138 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4jv\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.849572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.849542 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.850069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.850045 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.850262 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.850249 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.851012 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851211 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.851197 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851588 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.851521 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.850965 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.851290 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.851553 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.850365 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.851953 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.849987 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.857345 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.857316 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.887140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.887100 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:28.911286 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.911251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4jv\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:28.911743 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:28.911716 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\") pod \"e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:29.042752 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.042718 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.046822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.046800 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:29.196937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.196858 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.296974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.296945 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:47:29.338696 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.338672 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:47:29.358841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.358814 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\") pod \"30bc954d-59ed-4c85-9259-65b4898d0f61\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " Oct 31 00:47:29.359000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.358987 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\" (UniqueName: \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\") pod \"30bc954d-59ed-4c85-9259-65b4898d0f61\" (UID: \"30bc954d-59ed-4c85-9259-65b4898d0f61\") " Oct 31 00:47:29.359042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359026 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\") pod \"18c04fb5-2756-4931-828d-904438781bfa\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " Oct 31 00:47:29.359087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359072 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\" (UniqueName: \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\") pod \"18c04fb5-2756-4931-828d-904438781bfa\" (UID: \"18c04fb5-2756-4931-828d-904438781bfa\") " Oct 31 00:47:29.359139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359118 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume" pod "30bc954d-59ed-4c85-9259-65b4898d0f61" (UID: "30bc954d-59ed-4c85-9259-65b4898d0f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:29.359420 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359398 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume" pod "18c04fb5-2756-4931-828d-904438781bfa" (UID: "18c04fb5-2756-4931-828d-904438781bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:29.359680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359657 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bc954d-59ed-4c85-9259-65b4898d0f61-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:29.359680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.359681 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c04fb5-2756-4931-828d-904438781bfa-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:29.361105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.361084 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq" pod "30bc954d-59ed-4c85-9259-65b4898d0f61" (UID: "30bc954d-59ed-4c85-9259-65b4898d0f61"). InnerVolumeSpecName "kube-api-access-zqqcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:29.361181 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.361108 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82" pod "18c04fb5-2756-4931-828d-904438781bfa" (UID: "18c04fb5-2756-4931-828d-904438781bfa"). InnerVolumeSpecName "kube-api-access-lfr82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:29.460715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.460674 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqcq\" (UniqueName: \"kubernetes.io/projected/30bc954d-59ed-4c85-9259-65b4898d0f61-kube-api-access-zqqcq\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:29.460715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.460709 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfr82\" (UniqueName: \"kubernetes.io/projected/18c04fb5-2756-4931-828d-904438781bfa-kube-api-access-lfr82\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:29.493501 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:47:29.493467 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdac5ef_9fbe_4685_b550_04132790323c.slice/crio-fe7606d7352f4d29ab1de7809c45e927cc80f319d07dc01b8d0faf9d15666cfa WatchSource:0}: Error finding container fe7606d7352f4d29ab1de7809c45e927cc80f319d07dc01b8d0faf9d15666cfa: Status 404 returned error can't find the container with id fe7606d7352f4d29ab1de7809c45e927cc80f319d07dc01b8d0faf9d15666cfa Oct 31 00:47:29.496627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.496599 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:47:29.496727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.496703 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:29.496779 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.496762 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:29.496813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:29.496803 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:47:30.031564 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.031496 4471 generic.go:356] "Generic (PLEG): container finished" podID="18c04fb5-2756-4931-828d-904438781bfa" containerID="9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc" exitCode=0 Oct 31 00:47:30.032254 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.032225 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" event={"ID":"18c04fb5-2756-4931-828d-904438781bfa","Type":"ContainerDied","Data":"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc"} Oct 31 00:47:30.032421 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.032402 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" event={"ID":"18c04fb5-2756-4931-828d-904438781bfa","Type":"ContainerDied","Data":"903e4633ffda37dba073657fa65ea1b1ea108d4cfda95277ae0d3090d92d8183"} Oct 31 00:47:30.032551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.032524 4471 scope.go:122] "RemoveContainer" containerID="9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc" Oct 31 00:47:30.032848 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.032827 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt" Oct 31 00:47:30.044221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.044181 4471 generic.go:356] "Generic (PLEG): container finished" podID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerID="467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b" exitCode=0 Oct 31 00:47:30.044396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.044319 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" event={"ID":"30bc954d-59ed-4c85-9259-65b4898d0f61","Type":"ContainerDied","Data":"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b"} Oct 31 00:47:30.044396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.044371 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" event={"ID":"30bc954d-59ed-4c85-9259-65b4898d0f61","Type":"ContainerDied","Data":"67a8a65a1126fa40eb01c1766330e687a1eaf6142689b0cef2330c79edaea82b"} Oct 31 00:47:30.044509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.044463 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk" Oct 31 00:47:30.049744 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.049696 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" event={"ID":"4bdac5ef-9fbe-4685-b550-04132790323c","Type":"ContainerStarted","Data":"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e"} Oct 31 00:47:30.049921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.049777 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" event={"ID":"4bdac5ef-9fbe-4685-b550-04132790323c","Type":"ContainerStarted","Data":"fe7606d7352f4d29ab1de7809c45e927cc80f319d07dc01b8d0faf9d15666cfa"} Oct 31 00:47:30.055489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.055318 4471 scope.go:122] "RemoveContainer" containerID="9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc" Oct 31 00:47:30.055688 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:30.055668 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc\": container with ID starting with 9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc not found: ID does not exist" containerID="9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc" Oct 31 00:47:30.055739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.055701 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc"} err="failed to get container status \"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc\": rpc error: code = NotFound desc = could not find container \"9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc\": container with ID starting with 9f4d9f9cf088ad2a7de55637554fc8635d58f12617e77187c6dbe0b484efc6bc not found: ID does not exist" Oct 31 00:47:30.055739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.055718 4471 scope.go:122] "RemoveContainer" containerID="467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b" Oct 31 00:47:30.088853 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.088822 4471 scope.go:122] "RemoveContainer" containerID="467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b" Oct 31 00:47:30.089426 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:30.089396 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b\": container with ID starting with 467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b not found: ID does not exist" containerID="467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b" Oct 31 00:47:30.089542 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.089442 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b"} err="failed to get container status \"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b\": rpc error: code = NotFound desc = could not find container \"467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b\": container with ID starting with 467f13d1ae1a202627c7eb3af201c09af9dcad8f836571e186ac8989d623120b not found: ID does not exist" Oct 31 00:47:30.207290 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.207253 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:47:30.213709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.213688 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7d5f4de7-ffd6698f9-nwczt"] Oct 31 00:47:30.351086 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.351035 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podStartSLOduration=2.351017628 podStartE2EDuration="2.351017628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:30.349018627 +0000 UTC m=+1977.274287578" watchObservedRunningTime="2026-10-31 00:47:30.351017628 +0000 UTC m=+1977.276286580" Oct 31 00:47:30.493637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.492862 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:47:30.497380 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:30.497344 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7d5f4de7-cf6c94fdc-c7jqk"] Oct 31 00:47:31.048163 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:31.047960 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:31.051067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:31.050899 4471 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:47:31.051067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:31.050951 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:47:31.212355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:31.212320 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c04fb5-2756-4931-828d-904438781bfa" path="/var/lib/kubelet/pods/18c04fb5-2756-4931-828d-904438781bfa/volumes" Oct 31 00:47:31.212912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:31.212894 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" path="/var/lib/kubelet/pods/30bc954d-59ed-4c85-9259-65b4898d0f61/volumes" Oct 31 00:47:32.058556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:32.048907 4471 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:47:32.058556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:32.048966 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:47:32.395550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:32.393870 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:32.583100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:32.583058 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:32.731898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:32.731796 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:33.047506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:33.047423 4471 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:47:33.047506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:33.047480 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:47:34.048162 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:34.048080 4471 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:47:34.048162 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:34.048131 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:47:34.300120 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:34.300028 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:47:34.503191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:34.503154 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:35.047376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:35.047344 4471 patch_prober.go:29] interesting pod/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:47:35.047571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:35.047410 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 00:47:36.053208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:36.053176 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:36.053606 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:36.053585 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:36.054382 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:36.054364 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:47:37.299926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:37.299888 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:37.468023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:37.467989 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:39.835815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:39.835764 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:44.929797 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.929759 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:47:44.930567 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930545 4471 cpu_manager.go:414] "containerMap: removing container" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" Oct 31 00:47:44.930687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930569 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" Oct 31 00:47:44.930687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930584 4471 cpu_manager.go:414] "containerMap: removing container" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" Oct 31 00:47:44.930687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930592 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" Oct 31 00:47:44.930843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930788 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c04fb5-2756-4931-828d-904438781bfa" containerName="payload-pre-processing" Oct 31 00:47:44.930843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.930801 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bc954d-59ed-4c85-9259-65b4898d0f61" containerName="payload-processing" Oct 31 00:47:44.936035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.936010 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:44.956125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.956098 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:44.956246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.956164 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:44.956337 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.956320 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnkd\" (UniqueName: \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:44.956395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.956382 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:44.993670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.993639 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:47:44.995174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:44.995146 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c0f56c-serving-cert\"" Oct 31 00:47:45.057514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.057461 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnkd\" (UniqueName: \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.057740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.057595 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.057740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.057686 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.057855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.057749 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.057932 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:45.057909 4471 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-serving-cert: secret "maas-api-e2e-api-a-c0f56c-serving-cert" not found Oct 31 00:47:45.057995 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:47:45.057982 4471 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls podName:21635752-659e-4624-9d03-0d4b15d25c3e nodeName:}" failed. No retries permitted until 2026-10-31 00:47:45.55796152 +0000 UTC m=+1992.483230466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls") pod "maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" (UID: "21635752-659e-4624-9d03-0d4b15d25c3e") : secret "maas-api-e2e-api-a-c0f56c-serving-cert" not found Oct 31 00:47:45.058328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.058310 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.060109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.060091 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.120090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.120050 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnkd\" (UniqueName: \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.137373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.137310 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.295999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.295901 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.462742 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.462705 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:45.565414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.565310 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.567883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.567857 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") pod \"maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:45.850233 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:45.850154 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:46.229091 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:47:46.229058 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21635752_659e_4624_9d03_0d4b15d25c3e.slice/crio-07f154d9948965630bf45f7a7791b24700993b4e6b0c700591c24829af42ac26 WatchSource:0}: Error finding container 07f154d9948965630bf45f7a7791b24700993b4e6b0c700591c24829af42ac26: Status 404 returned error can't find the container with id 07f154d9948965630bf45f7a7791b24700993b4e6b0c700591c24829af42ac26 Oct 31 00:47:46.230873 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:46.230845 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:47:47.186326 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:47.186177 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" event={"ID":"21635752-659e-4624-9d03-0d4b15d25c3e","Type":"ContainerStarted","Data":"07f154d9948965630bf45f7a7791b24700993b4e6b0c700591c24829af42ac26"} Oct 31 00:47:48.193677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:48.193640 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" event={"ID":"21635752-659e-4624-9d03-0d4b15d25c3e","Type":"ContainerStarted","Data":"6f2d325a381f410c6f42a4429d07e72277da2e9c5ee83cad8ea14184d2009f32"} Oct 31 00:47:48.194152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:48.193703 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:48.313644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:48.313588 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" podStartSLOduration=3.407841086 podStartE2EDuration="4.313569321s" totalImagesPullingTime="905.728235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:48.313103963 +0000 UTC m=+1995.238372918" watchObservedRunningTime="2026-10-31 00:47:48.313569321 +0000 UTC m=+1995.238838293" Oct 31 00:47:50.248926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:50.248884 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:54.206173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:54.206138 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:47:55.307636 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:55.307581 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.474034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:55.473996 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:55.619898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:47:55.619816 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:00.814923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:00.814841 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.282437 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:48:04.282395 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:48:04.367014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:04.366981 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.517078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:04.517042 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:06.371147 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:06.371116 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.834540 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:11.834493 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.632414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:13.632323 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:13.776693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:13.776658 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:17.118451 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:17.118412 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:22.573441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:22.573351 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.336302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:27.336268 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:27.904790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:27.904740 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.552068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:28.552008 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:28.700826 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:28.700789 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:32.881884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:32.881846 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:34.285274 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:48:34.285233 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:48:35.486911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.486875 4471 generic.go:356] "Generic (PLEG): container finished" podID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerID="01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46" exitCode=0 Oct 31 00:48:35.487363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.486918 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerDied","Data":"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46"} Oct 31 00:48:35.487363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.487327 4471 scope.go:122] "RemoveContainer" containerID="01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46" Oct 31 00:48:35.488870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.488840 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:35.488968 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.488907 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:35.488968 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:35.488941 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:48:36.508418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:36.508373 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerStarted","Data":"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9"} Oct 31 00:48:37.117870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:37.117833 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:48:37.119301 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:37.119277 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:48:37.119417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:37.119325 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:48:37.781557 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:37.780686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:38.118090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:38.117998 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:48:38.118090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:38.118055 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:48:39.118480 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:39.118447 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:48:39.118908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:39.118508 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:48:40.118260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:40.118220 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:48:40.118447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:40.118281 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:48:40.393765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:40.393731 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:48:41.117792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:41.117749 4471 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 00:48:41.118026 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:41.117828 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 00:48:42.123686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:42.123652 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:48:42.125057 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:42.125035 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:48:42.666949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:42.666913 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:44.880087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:44.880048 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:45.028333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:45.028301 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.601316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:47.601275 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:52.689362 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:52.689321 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:57.825873 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:48:57.825831 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:03.080808 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:03.080758 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:03.780758 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:03.780723 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:49:03.780947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:03.780925 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" containerID="cri-o://038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6" gracePeriod=30 Oct 31 00:49:03.833641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:03.833603 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:49:03.833907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:03.833856 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" containerID="cri-o://1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632" gracePeriod=30 Oct 31 00:49:04.281349 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:04.281304 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:49:06.939402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:06.939358 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:06.939867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:06.939431 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:07.419007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:07.418959 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:49:07.419187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:07.419046 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:49:08.479122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:08.479087 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:08.641512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:08.641484 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:08.792887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:08.792454 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:09.120495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.120469 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:49:09.167391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.167365 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:49:09.186419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186383 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.186604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186453 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186486 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186602 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186667 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186693 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186731 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.186858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186764 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186832 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.186999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186869 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.186999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186926 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.186999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.186976 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187029 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187053 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.187142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187065 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.187142 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187070 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187184 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187201 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.187332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187234 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187332 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187266 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\") pod \"892d7291-94c7-4d14-819c-e80df40bd347\" (UID: \"892d7291-94c7-4d14-819c-e80df40bd347\") " Oct 31 00:49:09.187512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187347 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.187512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187424 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\") pod \"9c616615-724c-46dd-abf7-c5adbe371156\" (UID: \"9c616615-724c-46dd-abf7-c5adbe371156\") " Oct 31 00:49:09.188059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187959 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.188059 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.187988 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188273 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188297 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188312 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188327 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188341 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188355 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188308 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188328 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:09.188398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.188387 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:09.189068 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.189039 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.189565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.189501 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:09.190170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.190033 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "kube-api-access-g75nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:09.190170 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.190137 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.190587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.190559 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:09.190978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.190944 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "kube-api-access-nkj9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:09.191065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.191052 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:09.191430 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.191409 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo" pod "9c616615-724c-46dd-abf7-c5adbe371156" (UID: "9c616615-724c-46dd-abf7-c5adbe371156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:09.191798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.191771 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy" pod "892d7291-94c7-4d14-819c-e80df40bd347" (UID: "892d7291-94c7-4d14-819c-e80df40bd347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:09.290116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290076 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c616615-724c-46dd-abf7-c5adbe371156-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290117 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c616615-724c-46dd-abf7-c5adbe371156-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290133 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892d7291-94c7-4d14-819c-e80df40bd347-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290151 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75nt\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-kube-api-access-g75nt\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290177 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c616615-724c-46dd-abf7-c5adbe371156-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290189 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c616615-724c-46dd-abf7-c5adbe371156-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290198 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290207 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290215 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkj9h\" (UniqueName: \"kubernetes.io/projected/892d7291-94c7-4d14-819c-e80df40bd347-kube-api-access-nkj9h\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290229 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290240 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892d7291-94c7-4d14-819c-e80df40bd347-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.290322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.290248 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892d7291-94c7-4d14-819c-e80df40bd347-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:09.765549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.761456 4471 generic.go:356] "Generic (PLEG): container finished" podID="892d7291-94c7-4d14-819c-e80df40bd347" containerID="1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632" exitCode=0 Oct 31 00:49:09.765549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.761620 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" event={"ID":"892d7291-94c7-4d14-819c-e80df40bd347","Type":"ContainerDied","Data":"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632"} Oct 31 00:49:09.765549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.761673 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" event={"ID":"892d7291-94c7-4d14-819c-e80df40bd347","Type":"ContainerDied","Data":"b4018f9b05653954e3244012b23864891eed4a34606756863e72bca93fed7d7c"} Oct 31 00:49:09.765549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.761701 4471 scope.go:122] "RemoveContainer" containerID="1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632" Oct 31 00:49:09.765549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.761848 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw" Oct 31 00:49:09.778454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.778225 4471 generic.go:356] "Generic (PLEG): container finished" podID="9c616615-724c-46dd-abf7-c5adbe371156" containerID="038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6" exitCode=0 Oct 31 00:49:09.778454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.778348 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" event={"ID":"9c616615-724c-46dd-abf7-c5adbe371156","Type":"ContainerDied","Data":"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6"} Oct 31 00:49:09.778454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.778393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" event={"ID":"9c616615-724c-46dd-abf7-c5adbe371156","Type":"ContainerDied","Data":"5fcb531a40c9368130f6e9035748c365d9ed8f08b0f935e40cf997f026882823"} Oct 31 00:49:09.778694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.778473 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m" Oct 31 00:49:09.807401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.807107 4471 scope.go:122] "RemoveContainer" containerID="1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632" Oct 31 00:49:09.807594 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:09.807463 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632\": container with ID starting with 1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632 not found: ID does not exist" containerID="1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632" Oct 31 00:49:09.807594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.807503 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632"} err="failed to get container status \"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632\": rpc error: code = NotFound desc = could not find container \"1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632\": container with ID starting with 1bee1789486b3b8bee2a1d16dbac15178e8c0016cbe1a1080f1b050c39794632 not found: ID does not exist" Oct 31 00:49:09.807594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.807542 4471 scope.go:122] "RemoveContainer" containerID="038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6" Oct 31 00:49:09.830511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.830486 4471 scope.go:122] "RemoveContainer" containerID="038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6" Oct 31 00:49:09.831058 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:09.831021 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6\": container with ID starting with 038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6 not found: ID does not exist" containerID="038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6" Oct 31 00:49:09.831185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.831069 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6"} err="failed to get container status \"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6\": rpc error: code = NotFound desc = could not find container \"038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6\": container with ID starting with 038e05d46c4470c08d98e956e7793971d15cd442608e2403914913a4747d7fa6 not found: ID does not exist" Oct 31 00:49:09.948553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.947252 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:49:09.954029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:09.952676 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-l64nw"] Oct 31 00:49:10.068087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:10.068003 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:49:10.070815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:10.070786 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-f6p8m"] Oct 31 00:49:11.210102 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:11.210066 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892d7291-94c7-4d14-819c-e80df40bd347" path="/var/lib/kubelet/pods/892d7291-94c7-4d14-819c-e80df40bd347/volumes" Oct 31 00:49:11.210622 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:11.210607 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c616615-724c-46dd-abf7-c5adbe371156" path="/var/lib/kubelet/pods/9c616615-724c-46dd-abf7-c5adbe371156/volumes" Oct 31 00:49:13.973073 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:13.973038 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:14.181288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:14.181229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:14.327945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:14.327857 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:14.477101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:14.477066 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.730583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:19.727717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:21.557390 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:21.557355 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:21.704758 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:21.704265 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:24.421549 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:24.418085 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:25.682289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:25.682247 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:49:25.685928 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:25.685645 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" podUID="21635752-659e-4624-9d03-0d4b15d25c3e" containerName="maas-api" containerID="cri-o://6f2d325a381f410c6f42a4429d07e72277da2e9c5ee83cad8ea14184d2009f32" gracePeriod=30 Oct 31 00:49:25.793831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:25.793798 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:25.921519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:25.919075 4471 generic.go:356] "Generic (PLEG): container finished" podID="21635752-659e-4624-9d03-0d4b15d25c3e" containerID="6f2d325a381f410c6f42a4429d07e72277da2e9c5ee83cad8ea14184d2009f32" exitCode=0 Oct 31 00:49:25.921519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:25.919198 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" event={"ID":"21635752-659e-4624-9d03-0d4b15d25c3e","Type":"ContainerDied","Data":"6f2d325a381f410c6f42a4429d07e72277da2e9c5ee83cad8ea14184d2009f32"} Oct 31 00:49:26.134511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.134481 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:49:26.158069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.158030 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:26.305608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.305498 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\") pod \"21635752-659e-4624-9d03-0d4b15d25c3e\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " Oct 31 00:49:26.305790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.305631 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\" (UniqueName: \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\") pod \"21635752-659e-4624-9d03-0d4b15d25c3e\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " Oct 31 00:49:26.305790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.305663 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\") pod \"21635752-659e-4624-9d03-0d4b15d25c3e\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " Oct 31 00:49:26.305790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.305764 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") pod \"21635752-659e-4624-9d03-0d4b15d25c3e\" (UID: \"21635752-659e-4624-9d03-0d4b15d25c3e\") " Oct 31 00:49:26.307366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.307331 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca" pod "21635752-659e-4624-9d03-0d4b15d25c3e" (UID: "21635752-659e-4624-9d03-0d4b15d25c3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:26.309426 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.309396 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs" pod "21635752-659e-4624-9d03-0d4b15d25c3e" (UID: "21635752-659e-4624-9d03-0d4b15d25c3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:26.311637 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.311602 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd" pod "21635752-659e-4624-9d03-0d4b15d25c3e" (UID: "21635752-659e-4624-9d03-0d4b15d25c3e"). InnerVolumeSpecName "kube-api-access-rjnkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:26.313372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.313346 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls" pod "21635752-659e-4624-9d03-0d4b15d25c3e" (UID: "21635752-659e-4624-9d03-0d4b15d25c3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:26.408115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.407798 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.408115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.407836 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21635752-659e-4624-9d03-0d4b15d25c3e-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.408115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.407853 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnkd\" (UniqueName: \"kubernetes.io/projected/21635752-659e-4624-9d03-0d4b15d25c3e-kube-api-access-rjnkd\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.408115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.407868 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21635752-659e-4624-9d03-0d4b15d25c3e-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:26.451049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.451017 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:26.926522 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.926493 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" Oct 31 00:49:26.927037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.926561 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx" event={"ID":"21635752-659e-4624-9d03-0d4b15d25c3e","Type":"ContainerDied","Data":"07f154d9948965630bf45f7a7791b24700993b4e6b0c700591c24829af42ac26"} Oct 31 00:49:26.927037 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:26.926631 4471 scope.go:122] "RemoveContainer" containerID="6f2d325a381f410c6f42a4429d07e72277da2e9c5ee83cad8ea14184d2009f32" Oct 31 00:49:27.098284 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:27.098250 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:49:27.148792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:27.148757 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c0f56c-865db7bcfb-hp9gx"] Oct 31 00:49:27.211062 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:27.210986 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21635752-659e-4624-9d03-0d4b15d25c3e" path="/var/lib/kubelet/pods/21635752-659e-4624-9d03-0d4b15d25c3e/volumes" Oct 31 00:49:27.244202 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:27.244170 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:27.999153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:27.999074 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:28.144437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:28.144407 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:28.668394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:28.668353 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:28.816191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:28.816154 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:31.965814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.965783 4471 generic.go:356] "Generic (PLEG): container finished" podID="413e0848-de9b-428e-8c32-5cbd16504047" containerID="f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445" exitCode=0 Oct 31 00:49:31.966231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.965861 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerDied","Data":"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445"} Oct 31 00:49:31.966322 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.966309 4471 scope.go:122] "RemoveContainer" containerID="f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445" Oct 31 00:49:31.968077 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.968052 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:49:31.968150 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.968124 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:49:31.968187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:31.968156 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:49:32.988601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:32.988563 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerStarted","Data":"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4"} Oct 31 00:49:33.218359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.218332 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.382613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.380467 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.526619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.526443 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:49:33.527875 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.527839 4471 patch_prober.go:29] interesting pod/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:33.528038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.527905 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:33.533252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.533208 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:33.746978 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:33.746948 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:34.309068 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:34.309034 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:49:34.531560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:34.530895 4471 patch_prober.go:29] interesting pod/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:34.531560 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:34.530958 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:35.526355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:35.526316 4471 patch_prober.go:29] interesting pod/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:35.526797 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:35.526367 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:35.703030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:35.702982 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:35.862618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:35.859395 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:36.064554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:36.061905 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:36.089495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:36.089460 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:49:36.089748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:36.089719 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" containerID="cri-o://13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4" gracePeriod=30 Oct 31 00:49:38.058763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:38.055029 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:38.267120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:38.267082 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:38.935562 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:38.925426 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:39.286660 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:39.286573 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:39.437387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:39.437330 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:39.492856 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:39.492819 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:49:39.823291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:39.823255 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.099995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:40.099905 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:40.759078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:40.759024 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.048479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:41.048224 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.326411 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:41.326326 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:41.478025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:41.477986 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:43.658067 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.658041 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:49:43.793095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793010 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793095 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793062 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793109 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793129 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793194 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793277 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793331 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:43.793571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793407 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793437 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793493 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\") pod \"413e0848-de9b-428e-8c32-5cbd16504047\" (UID: \"413e0848-de9b-428e-8c32-5cbd16504047\") " Oct 31 00:49:43.793571 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793519 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:43.793838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793596 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:43.793838 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793704 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:43.793937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.793871 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:43.794152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.794133 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.794252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.794174 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.794252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.794189 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.794252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.794214 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/413e0848-de9b-428e-8c32-5cbd16504047-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.794252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.794229 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.795454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.795432 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:49:43.795638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.795615 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "kube-api-access-lsxt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:43.795841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.795822 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:43.795902 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.795840 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token" pod "413e0848-de9b-428e-8c32-5cbd16504047" (UID: "413e0848-de9b-428e-8c32-5cbd16504047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:43.895397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.895360 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxt7\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-kube-api-access-lsxt7\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.895397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.895394 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413e0848-de9b-428e-8c32-5cbd16504047-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.895397 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.895403 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/413e0848-de9b-428e-8c32-5cbd16504047-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:43.895673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:43.895412 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/413e0848-de9b-428e-8c32-5cbd16504047-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:44.287741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.287702 4471 generic.go:356] "Generic (PLEG): container finished" podID="413e0848-de9b-428e-8c32-5cbd16504047" containerID="13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4" exitCode=0 Oct 31 00:49:44.287959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.287756 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" Oct 31 00:49:44.287959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.287790 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerDied","Data":"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4"} Oct 31 00:49:44.287959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.287857 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5" event={"ID":"413e0848-de9b-428e-8c32-5cbd16504047","Type":"ContainerDied","Data":"9d82675d448e0cfb55bc2c72189d2f65cdada80b584be49ed39cd81deceddc05"} Oct 31 00:49:44.287959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.287886 4471 scope.go:122] "RemoveContainer" containerID="13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4" Oct 31 00:49:44.298728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.298711 4471 scope.go:122] "RemoveContainer" containerID="f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445" Oct 31 00:49:44.308629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.308604 4471 scope.go:122] "RemoveContainer" containerID="13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4" Oct 31 00:49:44.308924 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:44.308902 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4\": container with ID starting with 13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4 not found: ID does not exist" containerID="13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4" Oct 31 00:49:44.308965 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.308933 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4"} err="failed to get container status \"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4\": rpc error: code = NotFound desc = could not find container \"13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4\": container with ID starting with 13100065d2db7e3a6fc0115e09bcc9746c4ea0971308fda9c357789992904fe4 not found: ID does not exist" Oct 31 00:49:44.308965 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.308954 4471 scope.go:122] "RemoveContainer" containerID="f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445" Oct 31 00:49:44.309220 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:49:44.309203 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445\": container with ID starting with f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445 not found: ID does not exist" containerID="f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445" Oct 31 00:49:44.309262 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.309227 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445"} err="failed to get container status \"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445\": rpc error: code = NotFound desc = could not find container \"f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445\": container with ID starting with f34d5241c7734f72945115b38a8b6db9006dc88e251911cc8faefbe50d981445 not found: ID does not exist" Oct 31 00:49:44.455949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.455919 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:49:44.506597 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.506555 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c0f56c-openshift-default-665cfcdf75-fbgz5"] Oct 31 00:49:44.551157 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.551076 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:44.702804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:44.702772 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.210728 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:45.210576 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413e0848-de9b-428e-8c32-5cbd16504047" path="/var/lib/kubelet/pods/413e0848-de9b-428e-8c32-5cbd16504047/volumes" Oct 31 00:49:45.681075 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:45.681022 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.547333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:46.547301 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.693054 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:46.693018 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.682557 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:47.681117 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:47.841968 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:47.841932 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:48.079112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:48.079016 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:49.861905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:49.861873 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.010401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:50.010366 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.185625 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:50.181663 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.600474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:50.598713 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:50.799912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:50.799879 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:52.330829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:52.330790 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:54.795383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:54.795345 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:54.850677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:54.850638 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:49:55.034300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:55.034259 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:55.197394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:55.197355 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:57.822579 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:49:57.822539 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.589384 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:00.589344 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:00.741188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:00.741147 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.425205 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:03.425162 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:04.282114 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:04.282082 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:50:04.287144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:04.287117 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:50:04.639083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:04.638997 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:04.915146 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:04.915115 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:08.775919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:08.775881 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.288034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.287994 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" podUID="4f75838b-8bc3-4a19-9584-ddb728537e41" containerName="payload-processing" containerID="cri-o://cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3" gracePeriod=25 Oct 31 00:50:09.551311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.551246 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:50:09.592713 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.592680 4471 generic.go:356] "Generic (PLEG): container finished" podID="4f75838b-8bc3-4a19-9584-ddb728537e41" containerID="cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3" exitCode=0 Oct 31 00:50:09.592887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.592720 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" Oct 31 00:50:09.592887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.592744 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" event={"ID":"4f75838b-8bc3-4a19-9584-ddb728537e41","Type":"ContainerDied","Data":"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3"} Oct 31 00:50:09.592887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.592786 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb" event={"ID":"4f75838b-8bc3-4a19-9584-ddb728537e41","Type":"ContainerDied","Data":"2efc31a2409dd3c87562e1a635ee4242914b0de7edb1713b625c4bb1200ba426"} Oct 31 00:50:09.592887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.592809 4471 scope.go:122] "RemoveContainer" containerID="cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3" Oct 31 00:50:09.602222 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.602203 4471 scope.go:122] "RemoveContainer" containerID="cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3" Oct 31 00:50:09.602497 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:09.602476 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3\": container with ID starting with cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3 not found: ID does not exist" containerID="cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3" Oct 31 00:50:09.602594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.602506 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3"} err="failed to get container status \"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3\": rpc error: code = NotFound desc = could not find container \"cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3\": container with ID starting with cedf807457ce744b32aa69632d0c2e1b0116af98ef780a52339ae6ec87c816c3 not found: ID does not exist" Oct 31 00:50:09.687914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.687880 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\" (UniqueName: \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\") pod \"4f75838b-8bc3-4a19-9584-ddb728537e41\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " Oct 31 00:50:09.688126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.688114 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\") pod \"4f75838b-8bc3-4a19-9584-ddb728537e41\" (UID: \"4f75838b-8bc3-4a19-9584-ddb728537e41\") " Oct 31 00:50:09.688577 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.688543 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume" pod "4f75838b-8bc3-4a19-9584-ddb728537e41" (UID: "4f75838b-8bc3-4a19-9584-ddb728537e41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.690166 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.690140 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7" pod "4f75838b-8bc3-4a19-9584-ddb728537e41" (UID: "4f75838b-8bc3-4a19-9584-ddb728537e41"). InnerVolumeSpecName "kube-api-access-l94f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.789827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.789785 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f75838b-8bc3-4a19-9584-ddb728537e41-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.789827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.789821 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l94f7\" (UniqueName: \"kubernetes.io/projected/4f75838b-8bc3-4a19-9584-ddb728537e41-kube-api-access-l94f7\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.871278 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:09.871178 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerName="main" containerID="cri-o://6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f" gracePeriod=45 Oct 31 00:50:10.069749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.069687 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:50:10.120479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.120451 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-982ae3-6c4b5dc9b9-mmctb"] Oct 31 00:50:10.123331 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.123280 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:50:10.297609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297576 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.297785 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297682 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.297841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297784 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\" (UniqueName: \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.297841 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297824 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.297934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297917 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.297987 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297946 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\" (UniqueName: \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\") pod \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\" (UID: \"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4\") " Oct 31 00:50:10.298135 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.297907 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.298575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.298269 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.298575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.298562 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.298729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.298586 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.299005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.298982 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.300387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.300358 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.300489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.300427 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.300489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.300453 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27" pod "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" (UID: "93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4"). InnerVolumeSpecName "kube-api-access-nxc27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.400400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.400362 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxc27\" (UniqueName: \"kubernetes.io/projected/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-kube-api-access-nxc27\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.400400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.400391 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.400400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.400400 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.400400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.400410 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.601020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.600986 4471 generic.go:356] "Generic (PLEG): container finished" podID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerID="6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f" exitCode=0 Oct 31 00:50:10.601197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.601030 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" Oct 31 00:50:10.601197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.601068 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" event={"ID":"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4","Type":"ContainerDied","Data":"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f"} Oct 31 00:50:10.601197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.601110 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf" event={"ID":"93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4","Type":"ContainerDied","Data":"edf1f79185dfbbfe6951344efbcd0615b049d195fdaa6928c52914d18e1743ab"} Oct 31 00:50:10.601197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.601128 4471 scope.go:122] "RemoveContainer" containerID="6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f" Oct 31 00:50:10.610682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.610665 4471 scope.go:122] "RemoveContainer" containerID="6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f" Oct 31 00:50:10.610949 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:10.610928 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f\": container with ID starting with 6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f not found: ID does not exist" containerID="6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f" Oct 31 00:50:10.610994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.610960 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f"} err="failed to get container status \"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f\": rpc error: code = NotFound desc = could not find container \"6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f\": container with ID starting with 6310d6731354f77b0517383dbc82d1c4f54b9de9ecac17893ebe0d0ed59e699f not found: ID does not exist" Oct 31 00:50:10.769191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.769101 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:50:10.821423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:10.821381 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-982ae3/simulator-w1-982ae3-kserve-54744df6d8-tspnf"] Oct 31 00:50:11.210248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:11.210214 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f75838b-8bc3-4a19-9584-ddb728537e41" path="/var/lib/kubelet/pods/4f75838b-8bc3-4a19-9584-ddb728537e41/volumes" Oct 31 00:50:11.210653 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:11.210640 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" path="/var/lib/kubelet/pods/93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4/volumes" Oct 31 00:50:14.114909 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:14.114872 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:19.476808 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:19.476772 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:21.707643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:21.707575 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:24.499240 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:24.499182 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:28.016090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:28.015959 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:28.291335 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:28.291234 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:28.798818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:28.798726 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:30.368706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:30.368674 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:30.599489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:30.599455 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:30.744387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:30.744356 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:31.322492 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:31.322460 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:31.841359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:31.841323 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.546935 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:32.546892 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:32.795228 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:32.795189 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.145388 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:34.145349 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:34.285145 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:34.285109 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:50:35.997794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:35.997758 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:37.485364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:37.485322 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:37.648482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:37.648429 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:37.876636 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:37.876546 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:50:37.876816 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:37.876792 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" containerID="cri-o://e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286" gracePeriod=30 Oct 31 00:50:37.893711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:37.893681 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:39.869827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:39.869790 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:40.019612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:40.019572 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:41.392292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:41.392232 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.136842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.136814 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:50:43.272014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.271915 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.271961 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272017 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272042 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272100 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272173 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272298 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272208 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272324 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272350 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:43.272502 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272380 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\") pod \"cbe7a287-b617-42f4-b531-a891fe920719\" (UID: \"cbe7a287-b617-42f4-b531-a891fe920719\") " Oct 31 00:50:43.272680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272620 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:43.273407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.272728 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:43.273407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273026 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:43.273407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273148 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:43.273764 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273714 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.273971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273927 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.273971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273954 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.273971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273969 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.274176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.273983 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbe7a287-b617-42f4-b531-a891fe920719-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.274688 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.274659 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:43.275013 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.274998 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:43.275252 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.275232 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "kube-api-access-dmhnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:43.275450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.275429 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token" pod "cbe7a287-b617-42f4-b531-a891fe920719" (UID: "cbe7a287-b617-42f4-b531-a891fe920719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:43.375995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.375927 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbe7a287-b617-42f4-b531-a891fe920719-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.375995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.375958 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbe7a287-b617-42f4-b531-a891fe920719-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.375995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.375968 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmhnq\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-kube-api-access-dmhnq\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.375995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.375983 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbe7a287-b617-42f4-b531-a891fe920719-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:43.457977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.457941 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.602999 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.602907 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:43.840659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.840625 4471 generic.go:356] "Generic (PLEG): container finished" podID="cbe7a287-b617-42f4-b531-a891fe920719" containerID="e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286" exitCode=0 Oct 31 00:50:43.840827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.840678 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" Oct 31 00:50:43.840827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.840682 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerDied","Data":"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286"} Oct 31 00:50:43.840827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.840792 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj" event={"ID":"cbe7a287-b617-42f4-b531-a891fe920719","Type":"ContainerDied","Data":"afec8ab83d602f5d70c09d49c46164672bc74e4a1c4028f586a589ba8d7ebd58"} Oct 31 00:50:43.840827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.840814 4471 scope.go:122] "RemoveContainer" containerID="e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286" Oct 31 00:50:43.851725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.851671 4471 scope.go:122] "RemoveContainer" containerID="f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89" Oct 31 00:50:43.860859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.860840 4471 scope.go:122] "RemoveContainer" containerID="e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286" Oct 31 00:50:43.861144 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:43.861127 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286\": container with ID starting with e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286 not found: ID does not exist" containerID="e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286" Oct 31 00:50:43.861185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.861153 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286"} err="failed to get container status \"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286\": rpc error: code = NotFound desc = could not find container \"e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286\": container with ID starting with e7fe38e32bec6a11bdd4ed65199a389983afd644b80ba9de217411e40993d286 not found: ID does not exist" Oct 31 00:50:43.861185 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.861171 4471 scope.go:122] "RemoveContainer" containerID="f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89" Oct 31 00:50:43.861409 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:43.861388 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89\": container with ID starting with f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89 not found: ID does not exist" containerID="f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89" Oct 31 00:50:43.861478 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:43.861419 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89"} err="failed to get container status \"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89\": rpc error: code = NotFound desc = could not find container \"f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89\": container with ID starting with f30716ec1446a7f0a0b7020e46e08ff5398751872cf9c8a244d0914e8ca6bb89 not found: ID does not exist" Oct 31 00:50:44.009235 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:44.009196 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:50:44.060312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:44.060275 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f32ec2-openshift-default-577fcdbf77-zd2vj"] Oct 31 00:50:45.209552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:45.209504 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe7a287-b617-42f4-b531-a891fe920719" path="/var/lib/kubelet/pods/cbe7a287-b617-42f4-b531-a891fe920719/volumes" Oct 31 00:50:46.669694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:46.669649 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:50.157202 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:50.157164 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:50:50.501564 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:50.501473 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:50.646707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:50.646670 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:51.387400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:51.387357 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.246352 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:53.246282 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.447191 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:53.447153 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.598905 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:53.598814 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:54.292221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:54.292179 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:55.157641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.157582 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" podUID="a9d97a53-f3b2-46d1-b89b-86925338f95d" containerName="payload-processing" containerID="cri-o://a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068" gracePeriod=25 Oct 31 00:50:55.437212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.437184 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:50:55.566916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.566885 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\" (UniqueName: \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\") pod \"a9d97a53-f3b2-46d1-b89b-86925338f95d\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " Oct 31 00:50:55.567102 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.566956 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\") pod \"a9d97a53-f3b2-46d1-b89b-86925338f95d\" (UID: \"a9d97a53-f3b2-46d1-b89b-86925338f95d\") " Oct 31 00:50:55.567385 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.567361 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume" pod "a9d97a53-f3b2-46d1-b89b-86925338f95d" (UID: "a9d97a53-f3b2-46d1-b89b-86925338f95d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:55.567608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.567591 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d97a53-f3b2-46d1-b89b-86925338f95d-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:55.569093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.569069 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn" pod "a9d97a53-f3b2-46d1-b89b-86925338f95d" (UID: "a9d97a53-f3b2-46d1-b89b-86925338f95d"). InnerVolumeSpecName "kube-api-access-27csn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:55.668629 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.668557 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27csn\" (UniqueName: \"kubernetes.io/projected/a9d97a53-f3b2-46d1-b89b-86925338f95d-kube-api-access-27csn\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:55.917755 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.917715 4471 generic.go:356] "Generic (PLEG): container finished" podID="a9d97a53-f3b2-46d1-b89b-86925338f95d" containerID="a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068" exitCode=0 Oct 31 00:50:55.917755 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.917756 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" Oct 31 00:50:55.918020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.917796 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" event={"ID":"a9d97a53-f3b2-46d1-b89b-86925338f95d","Type":"ContainerDied","Data":"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068"} Oct 31 00:50:55.918020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.917854 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5" event={"ID":"a9d97a53-f3b2-46d1-b89b-86925338f95d","Type":"ContainerDied","Data":"f39375d68e43c4d1641e55104c0941cf49ba1566fe7e23cb38a00b6c132881d8"} Oct 31 00:50:55.918020 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.917885 4471 scope.go:122] "RemoveContainer" containerID="a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068" Oct 31 00:50:55.929323 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.929305 4471 scope.go:122] "RemoveContainer" containerID="a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068" Oct 31 00:50:55.929609 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:50:55.929586 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068\": container with ID starting with a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068 not found: ID does not exist" containerID="a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068" Oct 31 00:50:55.929721 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:55.929615 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068"} err="failed to get container status \"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068\": rpc error: code = NotFound desc = could not find container \"a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068\": container with ID starting with a1c765c57d56314690159649a427d9b401f5e4e676c66b943f85355bf2086068 not found: ID does not exist" Oct 31 00:50:56.086398 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:56.086363 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:50:56.137010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:56.136973 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e9c3c1-57dd846c75-n5td5"] Oct 31 00:50:57.209323 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:57.209287 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d97a53-f3b2-46d1-b89b-86925338f95d" path="/var/lib/kubelet/pods/a9d97a53-f3b2-46d1-b89b-86925338f95d/volumes" Oct 31 00:50:58.674758 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:50:58.674721 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.495860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:00.495823 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.648804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:00.648758 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:00.866697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:00.866621 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:03.955165 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:03.955133 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.119231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:04.119184 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.282749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:04.282611 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:04.294349 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:04.294291 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:51:08.783325 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:08.783277 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.437569 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:10.437510 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.607225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:10.607185 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.807991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:10.807875 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:10.961907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:10.961871 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.115373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:11.115260 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:11.531644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:11.531608 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:13.182280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:13.182244 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:51:13.292343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:13.292045 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:51:13.532653 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:13.532525 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:13.681993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:13.681958 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.701783 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:15.701747 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.862176 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:15.862140 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.053200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:16.053116 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.240195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:16.240161 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.182659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.182620 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" podUID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" containerName="payload-processing" containerID="cri-o://c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a" gracePeriod=25 Oct 31 00:51:18.254490 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.254458 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.293350 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.293318 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" podUID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" containerName="payload-pre-processing" containerID="cri-o://79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d" gracePeriod=25 Oct 31 00:51:18.400548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.400508 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.459180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.459154 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:51:18.495646 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.495598 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\" (UniqueName: \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\") pod \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " Oct 31 00:51:18.495848 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.495732 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\") pod \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\" (UID: \"594859f6-ff0c-43c0-87b3-3c46e2ec37d1\") " Oct 31 00:51:18.496476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.496442 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume" pod "594859f6-ff0c-43c0-87b3-3c46e2ec37d1" (UID: "594859f6-ff0c-43c0-87b3-3c46e2ec37d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:18.498441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.498408 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb" pod "594859f6-ff0c-43c0-87b3-3c46e2ec37d1" (UID: "594859f6-ff0c-43c0-87b3-3c46e2ec37d1"). InnerVolumeSpecName "kube-api-access-cv6qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:18.566009 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.565977 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:51:18.597027 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.596986 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\" (UniqueName: \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\") pod \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " Oct 31 00:51:18.597309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.597288 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\") pod \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\" (UID: \"94c89e7e-e2bb-423a-b9fb-8c67a6219191\") " Oct 31 00:51:18.597995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.597969 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6qb\" (UniqueName: \"kubernetes.io/projected/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-kube-api-access-cv6qb\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:18.598113 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.597997 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594859f6-ff0c-43c0-87b3-3c46e2ec37d1-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:18.598255 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.598229 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume" pod "94c89e7e-e2bb-423a-b9fb-8c67a6219191" (UID: "94c89e7e-e2bb-423a-b9fb-8c67a6219191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:18.599914 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.599855 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc" pod "94c89e7e-e2bb-423a-b9fb-8c67a6219191" (UID: "94c89e7e-e2bb-423a-b9fb-8c67a6219191"). InnerVolumeSpecName "kube-api-access-595bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:18.609038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.609002 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:18.725613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.716493 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94c89e7e-e2bb-423a-b9fb-8c67a6219191-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:18.725613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:18.716551 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595bc\" (UniqueName: \"kubernetes.io/projected/94c89e7e-e2bb-423a-b9fb-8c67a6219191-kube-api-access-595bc\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:19.106820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.106731 4471 generic.go:356] "Generic (PLEG): container finished" podID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" containerID="c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a" exitCode=0 Oct 31 00:51:19.106820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.106807 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" Oct 31 00:51:19.107044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.106819 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" event={"ID":"594859f6-ff0c-43c0-87b3-3c46e2ec37d1","Type":"ContainerDied","Data":"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a"} Oct 31 00:51:19.107044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.106897 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc" event={"ID":"594859f6-ff0c-43c0-87b3-3c46e2ec37d1","Type":"ContainerDied","Data":"22de08f5d2d13deec1750a726dfb47708fa1ccf2b67bd85a19cedec01cb615d5"} Oct 31 00:51:19.107044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.106919 4471 scope.go:122] "RemoveContainer" containerID="c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a" Oct 31 00:51:19.108311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.108286 4471 generic.go:356] "Generic (PLEG): container finished" podID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" containerID="79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d" exitCode=0 Oct 31 00:51:19.108432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.108345 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" Oct 31 00:51:19.108432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.108354 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" event={"ID":"94c89e7e-e2bb-423a-b9fb-8c67a6219191","Type":"ContainerDied","Data":"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d"} Oct 31 00:51:19.108432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.108393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb" event={"ID":"94c89e7e-e2bb-423a-b9fb-8c67a6219191","Type":"ContainerDied","Data":"a61f3919418322c982f47058022147db0ebc4aff1ba73f387bd5febe959598d6"} Oct 31 00:51:19.117109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.117084 4471 scope.go:122] "RemoveContainer" containerID="c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a" Oct 31 00:51:19.117565 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:19.117379 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a\": container with ID starting with c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a not found: ID does not exist" containerID="c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a" Oct 31 00:51:19.117565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.117414 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a"} err="failed to get container status \"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a\": rpc error: code = NotFound desc = could not find container \"c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a\": container with ID starting with c04fb06792e9cf95e40228b40a944c90465cdc2e4b6594ddad6a077741b1aa8a not found: ID does not exist" Oct 31 00:51:19.117565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.117438 4471 scope.go:122] "RemoveContainer" containerID="79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d" Oct 31 00:51:19.129195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.129176 4471 scope.go:122] "RemoveContainer" containerID="79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d" Oct 31 00:51:19.129688 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:19.129668 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d\": container with ID starting with 79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d not found: ID does not exist" containerID="79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d" Oct 31 00:51:19.129757 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.129699 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d"} err="failed to get container status \"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d\": rpc error: code = NotFound desc = could not find container \"79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d\": container with ID starting with 79a5fc32f79b796693ac4b2774ac9431a7148330782445a71a9142f474469a0d not found: ID does not exist" Oct 31 00:51:19.272494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.272461 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:51:19.321690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.321657 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2594a9-6f4d5b64dc-s7mzb"] Oct 31 00:51:19.385105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.385019 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:51:19.388456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.388425 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2594a9-54c85c6687-lnthc"] Oct 31 00:51:19.596061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:19.596022 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.209029 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:21.208994 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" path="/var/lib/kubelet/pods/594859f6-ff0c-43c0-87b3-3c46e2ec37d1/volumes" Oct 31 00:51:21.209648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:21.209628 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" path="/var/lib/kubelet/pods/94c89e7e-e2bb-423a-b9fb-8c67a6219191/volumes" Oct 31 00:51:21.917402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:21.917367 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:24.557303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:24.557265 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:51:24.607062 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:24.607016 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:51:24.609777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:24.609742 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:51:24.752310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:24.752235 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:27.416143 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:27.416110 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:28.359538 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:28.359493 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.081605 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:32.068085 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.228009 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:32.227978 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:32.787239 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:32.787203 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:34.194161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:34.194122 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:34.332840 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:34.332761 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:51:34.891440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:34.891399 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:51:34.891717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:34.891683 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" containerID="cri-o://f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e" gracePeriod=30 Oct 31 00:51:35.226694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.226665 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:51:35.241188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.241156 4471 generic.go:356] "Generic (PLEG): container finished" podID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerID="f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e" exitCode=0 Oct 31 00:51:35.241357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.241214 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" Oct 31 00:51:35.241357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.241233 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" event={"ID":"81f21c62-f7c3-416c-89b7-ab2ee2302f5a","Type":"ContainerDied","Data":"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e"} Oct 31 00:51:35.241357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.241304 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" event={"ID":"81f21c62-f7c3-416c-89b7-ab2ee2302f5a","Type":"ContainerDied","Data":"8e10d0adf8c44e3de541f97a9b625f0c69172c17e83027dffa168e8b44680c4b"} Oct 31 00:51:35.241357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.241326 4471 scope.go:122] "RemoveContainer" containerID="f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e" Oct 31 00:51:35.253437 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.253412 4471 scope.go:122] "RemoveContainer" containerID="f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e" Oct 31 00:51:35.254003 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:35.253976 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e\": container with ID starting with f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e not found: ID does not exist" containerID="f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e" Oct 31 00:51:35.254103 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.254017 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e"} err="failed to get container status \"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e\": rpc error: code = NotFound desc = could not find container \"f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e\": container with ID starting with f11c2eb05a0022e976a9a04c3310766eb2beb4ce32e1b26796c903fece0bfb6e not found: ID does not exist" Oct 31 00:51:35.348777 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.348740 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.348977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.348801 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.348977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.348838 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.348977 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.348924 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.349139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.349009 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.349139 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.349081 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.349259 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.349235 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\") pod \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\" (UID: \"81f21c62-f7c3-416c-89b7-ab2ee2302f5a\") " Oct 31 00:51:35.350439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.349617 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:35.350439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.349905 4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-ca-configmap\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.352394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.352366 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:35.354481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.353818 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:35.354481 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.354033 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "kube-api-access-cbtw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:35.356273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.356247 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:35.356396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.356373 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:35.387854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.387810 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts" pod "81f21c62-f7c3-416c-89b7-ab2ee2302f5a" (UID: "81f21c62-f7c3-416c-89b7-ab2ee2302f5a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451473 4471 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-csr-dns-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451498 4471 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-kubeconfig\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451508 4471 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-local-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451518 4471 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-cacerts\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451552 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbtw6\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-kube-api-access-cbtw6\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.451566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.451564 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f21c62-f7c3-416c-89b7-ab2ee2302f5a-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.683820 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.683776 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:51:35.687079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:35.687055 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb"] Oct 31 00:51:37.209711 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:37.209676 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" path="/var/lib/kubelet/pods/81f21c62-f7c3-416c-89b7-ab2ee2302f5a/volumes" Oct 31 00:51:38.100850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:38.100813 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:39.578639 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.578590 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerName="main" containerID="cri-o://b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77" gracePeriod=45 Oct 31 00:51:39.627087 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.627017 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" podUID="06e03405-6e4a-444d-9b6f-c661b814b948" containerName="main" containerID="cri-o://97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad" gracePeriod=45 Oct 31 00:51:39.628427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.628397 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" containerID="cri-o://9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9" gracePeriod=45 Oct 31 00:51:39.903601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.903574 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:51:39.912929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.912899 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.913083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.912963 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\" (UniqueName: \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.913083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.913076 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.913213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.913179 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.913315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.913300 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.913373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.913339 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\") pod \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\" (UID: \"2309d5f3-0749-4c51-98d3-5781cbb0f3f6\") " Oct 31 00:51:39.914358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.914324 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:39.915241 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.915213 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:39.915394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.915280 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:39.916428 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.916373 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:39.917408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.917382 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:39.917621 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.917600 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk" pod "2309d5f3-0749-4c51-98d3-5781cbb0f3f6" (UID: "2309d5f3-0749-4c51-98d3-5781cbb0f3f6"). InnerVolumeSpecName "kube-api-access-j6pjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:39.931160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.931133 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:51:39.942466 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:39.942442 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:51:40.014447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014413 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.014671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014484 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.014671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014518 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\" (UniqueName: \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.014671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014573 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.014671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014606 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014690 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014769 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014796 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014823 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014876 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.014901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014889 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\" (UniqueName: \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014918 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014942 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\" (UniqueName: \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\") pod \"37541b57-3039-438c-bebf-f0f56a7db701\" (UID: \"37541b57-3039-438c-bebf-f0f56a7db701\") " Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.014967 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015029 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015045 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015086 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\" (UniqueName: \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\") pod \"06e03405-6e4a-444d-9b6f-c661b814b948\" (UID: \"06e03405-6e4a-444d-9b6f-c661b814b948\") " Oct 31 00:51:40.015216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015153 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.015723 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015702 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015729 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015744 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015759 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015773 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015788 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pjk\" (UniqueName: \"kubernetes.io/projected/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-kube-api-access-j6pjk\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015801 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015815 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015827 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015840 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015851 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2309d5f3-0749-4c51-98d3-5781cbb0f3f6-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.015900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.015863 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.017188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.017164 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.017786 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.017631 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:40.017786 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.017754 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:40.017948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.017792 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "kube-api-access-wmh6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:40.018431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.018393 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs" pod "37541b57-3039-438c-bebf-f0f56a7db701" (UID: "37541b57-3039-438c-bebf-f0f56a7db701"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:40.018716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.018694 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2" pod "06e03405-6e4a-444d-9b6f-c661b814b948" (UID: "06e03405-6e4a-444d-9b6f-c661b814b948"). InnerVolumeSpecName "kube-api-access-9btt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:40.117482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117448 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btt2\" (UniqueName: \"kubernetes.io/projected/06e03405-6e4a-444d-9b6f-c661b814b948-kube-api-access-9btt2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.117482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117480 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37541b57-3039-438c-bebf-f0f56a7db701-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.117482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117490 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37541b57-3039-438c-bebf-f0f56a7db701-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.117734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117498 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmh6q\" (UniqueName: \"kubernetes.io/projected/37541b57-3039-438c-bebf-f0f56a7db701-kube-api-access-wmh6q\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.117734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117509 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e03405-6e4a-444d-9b6f-c661b814b948-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.117734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.117516 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e03405-6e4a-444d-9b6f-c661b814b948-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:40.194238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.194203 4471 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-btpxb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:51:40.194406 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.194271 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-btpxb" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.85:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:51:40.275419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.275379 4471 generic.go:356] "Generic (PLEG): container finished" podID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerID="b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77" exitCode=0 Oct 31 00:51:40.275648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.275470 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" Oct 31 00:51:40.275648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.275522 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" event={"ID":"2309d5f3-0749-4c51-98d3-5781cbb0f3f6","Type":"ContainerDied","Data":"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77"} Oct 31 00:51:40.275648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.275599 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95" event={"ID":"2309d5f3-0749-4c51-98d3-5781cbb0f3f6","Type":"ContainerDied","Data":"7d470efe346bb40485704f762e7b4e74a935945e9e6f9e628267857ff33bd53c"} Oct 31 00:51:40.275648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.275627 4471 scope.go:122] "RemoveContainer" containerID="b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77" Oct 31 00:51:40.280799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.279178 4471 generic.go:356] "Generic (PLEG): container finished" podID="06e03405-6e4a-444d-9b6f-c661b814b948" containerID="97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad" exitCode=0 Oct 31 00:51:40.280799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.279301 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" event={"ID":"06e03405-6e4a-444d-9b6f-c661b814b948","Type":"ContainerDied","Data":"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad"} Oct 31 00:51:40.280799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.279338 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" event={"ID":"06e03405-6e4a-444d-9b6f-c661b814b948","Type":"ContainerDied","Data":"a40650888aed3830d49d57ca7280fe1337e5107dcd78b8deed95cc4e21d516ff"} Oct 31 00:51:40.280799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.279412 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l" Oct 31 00:51:40.291627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.291597 4471 scope.go:122] "RemoveContainer" containerID="b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77" Oct 31 00:51:40.292198 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.291593 4471 generic.go:356] "Generic (PLEG): container finished" podID="37541b57-3039-438c-bebf-f0f56a7db701" containerID="9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9" exitCode=0 Oct 31 00:51:40.292314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.291626 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" event={"ID":"37541b57-3039-438c-bebf-f0f56a7db701","Type":"ContainerDied","Data":"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9"} Oct 31 00:51:40.292379 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.292322 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" event={"ID":"37541b57-3039-438c-bebf-f0f56a7db701","Type":"ContainerDied","Data":"5a260edee248647c6dd9d7d97f303a505a1c6d4a41c624352e7234f81f1d4446"} Oct 31 00:51:40.292379 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.291689 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" Oct 31 00:51:40.292563 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:40.292483 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77\": container with ID starting with b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77 not found: ID does not exist" containerID="b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77" Oct 31 00:51:40.292645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.292517 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77"} err="failed to get container status \"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77\": rpc error: code = NotFound desc = could not find container \"b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77\": container with ID starting with b4c376ceae3b4f0df1aad5a812348db4771ece2d8ab20e6f3b328570a66ddd77 not found: ID does not exist" Oct 31 00:51:40.292645 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.292640 4471 scope.go:122] "RemoveContainer" containerID="97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad" Oct 31 00:51:40.305598 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.305570 4471 scope.go:122] "RemoveContainer" containerID="97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad" Oct 31 00:51:40.305988 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:40.305966 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad\": container with ID starting with 97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad not found: ID does not exist" containerID="97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad" Oct 31 00:51:40.306122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.306000 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad"} err="failed to get container status \"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad\": rpc error: code = NotFound desc = could not find container \"97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad\": container with ID starting with 97c2e24c61659e4d3f4358d104bd7135b6ac77fd4b6dfe40c1dfd125f94df2ad not found: ID does not exist" Oct 31 00:51:40.306122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.306028 4471 scope.go:122] "RemoveContainer" containerID="9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9" Oct 31 00:51:40.315508 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.315485 4471 scope.go:122] "RemoveContainer" containerID="9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9" Oct 31 00:51:40.315879 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:40.315845 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9\": container with ID starting with 9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9 not found: ID does not exist" containerID="9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9" Oct 31 00:51:40.315974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.315878 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9"} err="failed to get container status \"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9\": rpc error: code = NotFound desc = could not find container \"9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9\": container with ID starting with 9e43ce66f47094d6b7c8eeb241f3d4f59a56a600bf951b14b5d82e3c6e8294c9 not found: ID does not exist" Oct 31 00:51:40.443300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.443261 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:51:40.455712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.455620 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/distinct-2-w2-e8ba06-kserve-d85bb9f9d-sbb95"] Oct 31 00:51:40.565309 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.565275 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:51:40.569127 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.569102 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd"] Oct 31 00:51:40.678116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.678078 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:51:40.684417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.684385 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e8ba06/premium-w2-e8ba06-kserve-679d8f4bcd-vtc8l"] Oct 31 00:51:40.835044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:40.834941 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e8ba06/simulator-w2-e8ba06-kserve-5b4b5b86f5-64zxd" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 31 00:51:41.208950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:41.208906 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e03405-6e4a-444d-9b6f-c661b814b948" path="/var/lib/kubelet/pods/06e03405-6e4a-444d-9b6f-c661b814b948/volumes" Oct 31 00:51:41.209376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:41.209359 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" path="/var/lib/kubelet/pods/2309d5f3-0749-4c51-98d3-5781cbb0f3f6/volumes" Oct 31 00:51:41.209778 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:41.209761 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37541b57-3039-438c-bebf-f0f56a7db701" path="/var/lib/kubelet/pods/37541b57-3039-438c-bebf-f0f56a7db701/volumes" Oct 31 00:51:43.088310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:43.088275 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:43.902518 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:43.902480 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:51:43.902740 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:43.902712 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" containerID="cri-o://7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab" gracePeriod=30 Oct 31 00:51:48.088684 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:48.088627 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:48.363982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:48.363902 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-wspmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:51:48.363982 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:48.363972 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:51:49.164929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.164898 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:51:49.232614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232483 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232546 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232606 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232649 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232714 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232810 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232809 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232841 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.232896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232846 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:49.233178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232935 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.233178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232972 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:49.233178 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.232992 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\") pod \"2963405d-002b-4406-9002-27948db6bcb8\" (UID: \"2963405d-002b-4406-9002-27948db6bcb8\") " Oct 31 00:51:49.233311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233209 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:49.233311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233288 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:49.233521 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233500 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.233601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233557 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2963405d-002b-4406-9002-27948db6bcb8-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.233601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233576 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.233601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233591 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.233698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.233604 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.234980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.234951 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:51:49.234980 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.234970 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:49.235243 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.235222 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:49.235339 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.235239 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d" pod "2963405d-002b-4406-9002-27948db6bcb8" (UID: "2963405d-002b-4406-9002-27948db6bcb8"). InnerVolumeSpecName "kube-api-access-t5r8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:49.335464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.335344 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.335464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.335382 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5r8d\" (UniqueName: \"kubernetes.io/projected/2963405d-002b-4406-9002-27948db6bcb8-kube-api-access-t5r8d\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.335464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.335398 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2963405d-002b-4406-9002-27948db6bcb8-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.335464 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.335412 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2963405d-002b-4406-9002-27948db6bcb8-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:49.347792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.347754 4471 generic.go:356] "Generic (PLEG): container finished" podID="2963405d-002b-4406-9002-27948db6bcb8" containerID="7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab" exitCode=0 Oct 31 00:51:49.347985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.347796 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" Oct 31 00:51:49.347985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.347833 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" event={"ID":"2963405d-002b-4406-9002-27948db6bcb8","Type":"ContainerDied","Data":"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab"} Oct 31 00:51:49.347985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.347898 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm" event={"ID":"2963405d-002b-4406-9002-27948db6bcb8","Type":"ContainerDied","Data":"75bae741dbcc2e42cc87005504d01b6e947c338c2411243b8b8b7fa756f528cd"} Oct 31 00:51:49.347985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.347928 4471 scope.go:122] "RemoveContainer" containerID="7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab" Oct 31 00:51:49.358684 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.358665 4471 scope.go:122] "RemoveContainer" containerID="7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab" Oct 31 00:51:49.358961 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:51:49.358944 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab\": container with ID starting with 7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab not found: ID does not exist" containerID="7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab" Oct 31 00:51:49.359035 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.358978 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab"} err="failed to get container status \"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab\": rpc error: code = NotFound desc = could not find container \"7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab\": container with ID starting with 7a17847246c8cdd1414faf3e704aa61eaf19bf9ace0a3e747e4643ef6b46f3ab not found: ID does not exist" Oct 31 00:51:49.512799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.512711 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:51:49.516951 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:49.516929 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-wspmm"] Oct 31 00:51:51.208601 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:51.208564 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2963405d-002b-4406-9002-27948db6bcb8" path="/var/lib/kubelet/pods/2963405d-002b-4406-9002-27948db6bcb8/volumes" Oct 31 00:51:53.094015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.093965 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:53.580137 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580104 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:51:53.580607 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580592 4471 cpu_manager.go:414] "containerMap: removing container" podUID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" containerName="payload-pre-processing" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580608 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" containerName="payload-pre-processing" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580622 4471 cpu_manager.go:414] "containerMap: removing container" podUID="21635752-659e-4624-9d03-0d4b15d25c3e" containerName="maas-api" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580628 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21635752-659e-4624-9d03-0d4b15d25c3e" containerName="maas-api" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580637 4471 cpu_manager.go:414] "containerMap: removing container" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580642 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" Oct 31 00:51:53.580666 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580657 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580661 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580684 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580689 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580697 4471 cpu_manager.go:414] "containerMap: removing container" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580702 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580711 4471 cpu_manager.go:414] "containerMap: removing container" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580716 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580726 4471 cpu_manager.go:414] "containerMap: removing container" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580730 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580740 4471 cpu_manager.go:414] "containerMap: removing container" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580745 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580755 4471 cpu_manager.go:414] "containerMap: removing container" podUID="06e03405-6e4a-444d-9b6f-c661b814b948" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580759 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e03405-6e4a-444d-9b6f-c661b814b948" containerName="main" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580767 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a9d97a53-f3b2-46d1-b89b-86925338f95d" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580772 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d97a53-f3b2-46d1-b89b-86925338f95d" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580781 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4f75838b-8bc3-4a19-9584-ddb728537e41" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580785 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f75838b-8bc3-4a19-9584-ddb728537e41" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580792 4471 cpu_manager.go:414] "containerMap: removing container" podUID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580797 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" containerName="payload-processing" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580806 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580810 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580818 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580821 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580831 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" Oct 31 00:51:53.580834 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580835 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580908 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580915 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2963405d-002b-4406-9002-27948db6bcb8" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580921 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c616615-724c-46dd-abf7-c5adbe371156" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580927 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c89e7e-e2bb-423a-b9fb-8c67a6219191" containerName="payload-pre-processing" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580932 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="21635752-659e-4624-9d03-0d4b15d25c3e" containerName="maas-api" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580939 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="892d7291-94c7-4d14-819c-e80df40bd347" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580946 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2309d5f3-0749-4c51-98d3-5781cbb0f3f6" containerName="main" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580952 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe7a287-b617-42f4-b531-a891fe920719" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580958 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cd92b0-686d-47ba-9ab2-d3e72b6e0dd4" containerName="main" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580964 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580969 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f21c62-f7c3-416c-89b7-ab2ee2302f5a" containerName="discovery" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580975 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="37541b57-3039-438c-bebf-f0f56a7db701" containerName="main" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580980 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e03405-6e4a-444d-9b6f-c661b814b948" containerName="main" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580987 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d97a53-f3b2-46d1-b89b-86925338f95d" containerName="payload-processing" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580992 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f75838b-8bc3-4a19-9584-ddb728537e41" containerName="payload-processing" Oct 31 00:51:53.581546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.580997 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="594859f6-ff0c-43c0-87b3-3c46e2ec37d1" containerName="payload-processing" Oct 31 00:51:53.584200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.584185 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.684734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684694 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.684941 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684757 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.684941 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684835 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.684941 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684873 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.685119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684950 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.685119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.684992 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.685119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.685060 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.685119 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.685114 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62lx\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.685311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.685152 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786396 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786360 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786422 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786466 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786613 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62lx\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786689 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786857 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786833 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786906 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786880 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.786961 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.786943 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.787015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.787002 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.787510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.787481 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.791258 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.791224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.791434 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.791403 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:51:53.791817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.791793 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.792444 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.792414 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.792797 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.792773 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.794091 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.794069 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.795723 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.795701 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.853316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.853246 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.853440 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.853376 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62lx\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:53.896453 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:53.896415 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:54.319546 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:51:54.319504 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6908dfb5_e1aa_4e86_97ae_7b29dc14d3b4.slice/crio-4589d98142e2d65851012612432998b81e70edf139e2e3a35562c2955eb370f9 WatchSource:0}: Error finding container 4589d98142e2d65851012612432998b81e70edf139e2e3a35562c2955eb370f9: Status 404 returned error can't find the container with id 4589d98142e2d65851012612432998b81e70edf139e2e3a35562c2955eb370f9 Oct 31 00:51:54.321668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:54.321638 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:51:54.322414 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:54.322374 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:51:54.322489 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:54.322460 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:51:54.322559 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:54.322504 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:51:54.382352 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:54.382323 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" event={"ID":"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4","Type":"ContainerStarted","Data":"4589d98142e2d65851012612432998b81e70edf139e2e3a35562c2955eb370f9"} Oct 31 00:51:55.394619 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:55.394566 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" event={"ID":"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4","Type":"ContainerStarted","Data":"8c831e87ec2b335a835b1082aaf08582318a8e83371f3fb27cd3b4e3f82a16ff"} Oct 31 00:51:55.897303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:55.897272 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:51:55.898854 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:55.898827 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:51:55.898957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:55.898889 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:51:56.897019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:56.896976 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:51:56.897421 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:56.897038 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:51:57.443044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:57.442945 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:57.910554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:57.903303 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:51:57.910554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:57.903383 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:51:58.897224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:58.897184 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:51:58.897415 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:58.897245 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:51:59.030488 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:59.030450 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.282923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:59.282835 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.426685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:59.426653 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:59.897683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:59.897647 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 00:51:59.897881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:51:59.897717 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 00:52:00.012167 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.012137 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.822950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.819440 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:00.915872 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.915836 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:52:00.916919 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.916898 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:52:00.921791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.921766 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:52:00.969931 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:00.969898 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:01.087994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:01.087872 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podStartSLOduration=8.087859302 podStartE2EDuration="8.087859302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:55.578426478 +0000 UTC m=+2242.503695457" watchObservedRunningTime="2026-10-31 00:52:01.087859302 +0000 UTC m=+2248.013128371" Oct 31 00:52:02.760395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:02.760355 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:04.283893 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:04.283850 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:52:04.670371 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:04.670338 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.051806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:06.051774 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.197596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:06.197562 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.396468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:06.396437 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:06.400376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:06.400350 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:52:06.400594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:06.400569 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" containerID="cri-o://a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9" gracePeriod=30 Oct 31 00:52:07.479369 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:07.479336 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.648755 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:07.648715 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:07.845840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:07.845756 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:08.700110 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.700074 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:52:08.700614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.700595 4471 cpu_manager.go:414] "containerMap: removing container" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:52:08.700680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.700616 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:52:08.700747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.700738 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="413e0848-de9b-428e-8c32-5cbd16504047" containerName="istio-proxy" Oct 31 00:52:08.703260 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.703245 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.826823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.826789 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.826829 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.826854 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wzb\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.826971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.827015 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.827084 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.827131 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827199 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.827157 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.827302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.827228 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.909174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.909130 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:52:08.928790 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.928756 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.928972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.928876 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.928972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.928941 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.928972 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.928967 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.928991 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wzb\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929044 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929083 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929169 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929202 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929234 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929517 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929492 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929659 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929945 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.929993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.929978 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.931596 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.931578 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.931859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.931837 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.990311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.990240 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.990494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.990425 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wzb\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:08.998105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:08.998076 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.016462 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.016383 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:09.161213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.161174 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.310932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.310843 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.456545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.447995 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:52:09.461239 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:52:09.461198 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c5e6b5_4557_4aa1_b2b2_4ebdfbd4e6f8.slice/crio-e0784ceaf1efc0fab72d585286d335589fe7357e44aad8a7f1d77427891ab314 WatchSource:0}: Error finding container e0784ceaf1efc0fab72d585286d335589fe7357e44aad8a7f1d77427891ab314: Status 404 returned error can't find the container with id e0784ceaf1efc0fab72d585286d335589fe7357e44aad8a7f1d77427891ab314 Oct 31 00:52:09.468739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.468699 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:09.468860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.468786 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:09.468860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.468830 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:09.485947 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.485180 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" event={"ID":"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8","Type":"ContainerStarted","Data":"e0784ceaf1efc0fab72d585286d335589fe7357e44aad8a7f1d77427891ab314"} Oct 31 00:52:09.812244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.812179 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:09.983045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:09.980960 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:10.493024 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:10.492984 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" event={"ID":"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8","Type":"ContainerStarted","Data":"e83938c08b42093461ed4ecdcc98609660d8b83f633b95cc5e84f0683fc2a124"} Oct 31 00:52:10.614617 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:10.614558 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podStartSLOduration=2.614540572 podStartE2EDuration="2.614540572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:10.612773641 +0000 UTC m=+2257.538042624" watchObservedRunningTime="2026-10-31 00:52:10.614540572 +0000 UTC m=+2257.539809516" Oct 31 00:52:11.017638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.017597 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:11.019161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.019138 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:52:11.019231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.019189 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:52:11.222361 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.222327 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:11.598206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.598169 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:11.713221 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.713195 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:52:11.800706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800669 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.800910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800721 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.800910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800751 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.800910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800772 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.800910 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800878 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.801134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.800911 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.801134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801008 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.801134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801075 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.801134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801099 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:11.801134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801119 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\") pod \"debca05b-284b-4c74-a31a-ac5b7c271e2a\" (UID: \"debca05b-284b-4c74-a31a-ac5b7c271e2a\") " Oct 31 00:52:11.801392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801114 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:11.801392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801157 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:11.801392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801249 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:11.801599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801564 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:11.801599 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801586 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.801709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801601 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debca05b-284b-4c74-a31a-ac5b7c271e2a-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.801709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801610 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.801709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.801619 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.803116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.803088 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:11.803249 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.803132 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:11.803313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.803266 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:11.803375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.803355 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh" pod "debca05b-284b-4c74-a31a-ac5b7c271e2a" (UID: "debca05b-284b-4c74-a31a-ac5b7c271e2a"). InnerVolumeSpecName "kube-api-access-94qxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:11.902375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.902341 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.902375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.902372 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.902375 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.902384 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.902652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.902393 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debca05b-284b-4c74-a31a-ac5b7c271e2a-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.902652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.902404 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94qxh\" (UniqueName: \"kubernetes.io/projected/debca05b-284b-4c74-a31a-ac5b7c271e2a-kube-api-access-94qxh\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:11.919126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:11.919082 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:12.016867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.016784 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:52:12.016867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.016844 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:52:12.506100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.506063 4471 generic.go:356] "Generic (PLEG): container finished" podID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerID="a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9" exitCode=0 Oct 31 00:52:12.506613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.506126 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerDied","Data":"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9"} Oct 31 00:52:12.506613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.506170 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" Oct 31 00:52:12.506613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.506174 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx" event={"ID":"debca05b-284b-4c74-a31a-ac5b7c271e2a","Type":"ContainerDied","Data":"9a684331ffac53dc24c90ea89ed754ff9ac1a83421b402d848baab3862e4148c"} Oct 31 00:52:12.506613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.506183 4471 scope.go:122] "RemoveContainer" containerID="a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9" Oct 31 00:52:12.516267 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.516249 4471 scope.go:122] "RemoveContainer" containerID="01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46" Oct 31 00:52:12.525500 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.525324 4471 scope.go:122] "RemoveContainer" containerID="a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9" Oct 31 00:52:12.525649 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:12.525626 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9\": container with ID starting with a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9 not found: ID does not exist" containerID="a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9" Oct 31 00:52:12.525692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.525660 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9"} err="failed to get container status \"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9\": rpc error: code = NotFound desc = could not find container \"a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9\": container with ID starting with a93fdff2affbcdb3c49f7214cb5a168cba489d23323aec9db9fc12f2eecbb6b9 not found: ID does not exist" Oct 31 00:52:12.525692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.525678 4471 scope.go:122] "RemoveContainer" containerID="01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46" Oct 31 00:52:12.525946 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:12.525930 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46\": container with ID starting with 01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46 not found: ID does not exist" containerID="01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46" Oct 31 00:52:12.525993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.525952 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46"} err="failed to get container status \"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46\": rpc error: code = NotFound desc = could not find container \"01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46\": container with ID starting with 01ffa341ce234a67f02c10e92cfb10ce991ef5b728f6121c729ef69e38210d46 not found: ID does not exist" Oct 31 00:52:12.607443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.607400 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:12.701412 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.701374 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:52:12.705946 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:12.705921 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e9c3c1-openshift-default-77447d98f7-8v6vx"] Oct 31 00:52:13.017546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:13.017507 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:52:13.017736 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:13.017589 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:52:13.208509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:13.208477 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" path="/var/lib/kubelet/pods/debca05b-284b-4c74-a31a-ac5b7c271e2a/volumes" Oct 31 00:52:14.017282 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:14.017247 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:52:14.017741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:14.017304 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:52:15.017468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:15.017437 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 00:52:15.017907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:15.017488 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 00:52:16.021364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:16.021328 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:16.021782 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:16.021636 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:16.022369 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:16.022345 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:52:16.931715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:16.931672 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:18.832998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:18.832960 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:18.987471 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:18.987427 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:19.196701 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:19.196671 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.174196 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.174147 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.348220 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.348166 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:20.794419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.794387 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-407e80"] Oct 31 00:52:20.794893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.794880 4471 cpu_manager.go:414] "containerMap: removing container" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.794956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.794895 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.794956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.794926 4471 cpu_manager.go:414] "containerMap: removing container" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.794956 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.794932 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.795048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.795019 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.795048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.795030 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="debca05b-284b-4c74-a31a-ac5b7c271e2a" containerName="istio-proxy" Oct 31 00:52:20.799952 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.799926 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:20.852072 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.852039 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-407e80"] Oct 31 00:52:20.898722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.898688 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pdlhw\"" Oct 31 00:52:20.923302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:20.923268 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7gm\" (UniqueName: \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\") pod \"test-curl-7372-407e80\" (UID: \"cd1c119a-8bf5-4034-baab-d3526c0fdf73\") " pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:21.024875 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.024838 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7gm\" (UniqueName: \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\") pod \"test-curl-7372-407e80\" (UID: \"cd1c119a-8bf5-4034-baab-d3526c0fdf73\") " pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:21.086032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.085955 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7gm\" (UniqueName: \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\") pod \"test-curl-7372-407e80\" (UID: \"cd1c119a-8bf5-4034-baab-d3526c0fdf73\") " pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:21.111845 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.111803 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:21.496404 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:52:21.496369 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1c119a_8bf5_4034_baab_d3526c0fdf73.slice/crio-94ed8c1bf44685f1afa3b2e6d6dfe4c2a5940770ec1698f8fa29f5a4b3992f6b WatchSource:0}: Error finding container 94ed8c1bf44685f1afa3b2e6d6dfe4c2a5940770ec1698f8fa29f5a4b3992f6b: Status 404 returned error can't find the container with id 94ed8c1bf44685f1afa3b2e6d6dfe4c2a5940770ec1698f8fa29f5a4b3992f6b Oct 31 00:52:21.501203 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.501170 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-407e80"] Oct 31 00:52:21.507361 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.507345 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:52:21.560381 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:21.560340 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-407e80" event={"ID":"cd1c119a-8bf5-4034-baab-d3526c0fdf73","Type":"ContainerStarted","Data":"94ed8c1bf44685f1afa3b2e6d6dfe4c2a5940770ec1698f8fa29f5a4b3992f6b"} Oct 31 00:52:22.348388 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:22.348349 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:23.318386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.318339 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:23.801912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.801873 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:23.844463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.844423 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:52:23.848693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.848659 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.945314 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.945272 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967715 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967766 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967802 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967823 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967893 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5wm\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967924 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967954 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.967989 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:23.968214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:23.968006 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.069712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069636 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5wm\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.069712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069679 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.069934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069716 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.069934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069770 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.069934 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069794 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069933 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.069972 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070023 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070054 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070171 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070186 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070470 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070442 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070689 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.070792 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.070739 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.072632 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.072613 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.072945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.072924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.136991 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.136924 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.137152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.137051 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5wm\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\") pod \"e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.167266 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.167228 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:24.627237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.627199 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:52:24.656583 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:52:24.656541 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fbdefc_9c49_42e5_b842_2ac8bd269ba6.slice/crio-4540569ec6bb1988598bd5270e63cb10f476e35bb306e2dba638a5f21a174149 WatchSource:0}: Error finding container 4540569ec6bb1988598bd5270e63cb10f476e35bb306e2dba638a5f21a174149: Status 404 returned error can't find the container with id 4540569ec6bb1988598bd5270e63cb10f476e35bb306e2dba638a5f21a174149 Oct 31 00:52:24.660001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.659956 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:24.660108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.660057 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:24.660108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.660101 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:24.698681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.698645 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:52:24.712916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.712063 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.766152 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.766120 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:52:24.767490 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.767467 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bc2453-openshift-default-dockercfg-zk8sx\"" Oct 31 00:52:24.795759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.795718 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814198 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814252 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x27t\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814288 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814331 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814350 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814408 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814478 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814514 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.814825 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.814550 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916164 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916132 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916181 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916205 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916233 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916258 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x27t\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916288 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916322 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916342 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.916393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916390 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.917014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.916743 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.917091 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.917065 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.917765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.917738 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.917884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.917770 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.918073 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.918052 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.919457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.919434 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.919835 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.919814 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.997572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.997524 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x27t\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:24.999232 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:24.999206 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\") pod \"e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:25.036716 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.036680 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:25.064460 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.064379 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:25.502817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.501090 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:52:25.502817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.502366 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:25.502817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.502437 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:25.502817 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.502469 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:52:25.601275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.600511 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" event={"ID":"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6","Type":"ContainerStarted","Data":"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85"} Oct 31 00:52:25.601275 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.600592 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" event={"ID":"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6","Type":"ContainerStarted","Data":"4540569ec6bb1988598bd5270e63cb10f476e35bb306e2dba638a5f21a174149"} Oct 31 00:52:25.603763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.603657 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerStarted","Data":"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863"} Oct 31 00:52:25.603763 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.603709 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerStarted","Data":"5a9f2805b079227d1a1f31b87c72ab7effa17e928a66d5fd05bdfb1d04628d32"} Oct 31 00:52:25.722806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.722747 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podStartSLOduration=2.722729082 podStartE2EDuration="2.722729082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:25.72107698 +0000 UTC m=+2272.646345964" watchObservedRunningTime="2026-10-31 00:52:25.722729082 +0000 UTC m=+2272.647998038" Oct 31 00:52:25.845797 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:25.842377 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podStartSLOduration=1.84236144 podStartE2EDuration="1.84236144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:25.841755841 +0000 UTC m=+2272.767024794" watchObservedRunningTime="2026-10-31 00:52:25.84236144 +0000 UTC m=+2272.767630394" Oct 31 00:52:26.038030 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.037987 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:26.039200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.039173 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:26.039312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.039222 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:26.171554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.170903 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:26.178550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.176691 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:52:26.178550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.176764 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:52:26.614613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.614492 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-407e80" event={"ID":"cd1c119a-8bf5-4034-baab-d3526c0fdf73","Type":"ContainerStarted","Data":"46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa"} Oct 31 00:52:26.741197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:26.741142 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7372-407e80" podStartSLOduration=2.634977107 podStartE2EDuration="6.741127998s" totalImagesPullingTime="4.106150891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:26.739584269 +0000 UTC m=+2273.664853220" watchObservedRunningTime="2026-10-31 00:52:26.741127998 +0000 UTC m=+2273.666396950" Oct 31 00:52:27.041749 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.041715 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:27.182550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.182023 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:52:27.182550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.182088 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:52:27.183742 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.183707 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:27.350862 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.350756 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:27.620751 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.620668 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:27.621590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:27.621563 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:52:28.208281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.208244 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:52:28.211000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.210964 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:52:28.453151 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.453053 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-407e80"] Oct 31 00:52:28.485612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.453783 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-407e80"] Oct 31 00:52:28.625916 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.625875 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7372-407e80" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" containerName="test-curl-7372-407e80" containerID="cri-o://46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa" gracePeriod=2 Oct 31 00:52:28.681568 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:28.677861 4471 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7372-407e80\" is forbidden: User \"system:node:ip-10-0-134-11.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-11.us-west-2.compute.internal' and this object" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:28.713620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.713580 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:28.883313 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:28.883189 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:29.179555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:29.174957 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:52:29.179555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:29.175025 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:52:29.205111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:29.205072 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:29.352548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:29.352503 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:29.878551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:29.878066 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:30.032746 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.032715 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:30.179566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.179121 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 31 00:52:30.179566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.179183 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 31 00:52:30.206866 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.206833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:30.868281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.868251 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:30.940873 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.940837 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\" (UniqueName: \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\") pod \"cd1c119a-8bf5-4034-baab-d3526c0fdf73\" (UID: \"cd1c119a-8bf5-4034-baab-d3526c0fdf73\") " Oct 31 00:52:30.942974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:30.942947 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm" pod "cd1c119a-8bf5-4034-baab-d3526c0fdf73" (UID: "cd1c119a-8bf5-4034-baab-d3526c0fdf73"). InnerVolumeSpecName "kube-api-access-fz7gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:31.042830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.042789 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7gm\" (UniqueName: \"kubernetes.io/projected/cd1c119a-8bf5-4034-baab-d3526c0fdf73-kube-api-access-fz7gm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:31.156126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.156088 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:31.196520 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.196425 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:31.197012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.196988 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:31.197771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.197750 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:52:31.209475 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.209443 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" path="/var/lib/kubelet/pods/cd1c119a-8bf5-4034-baab-d3526c0fdf73/volumes" Oct 31 00:52:31.464187 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.464049 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:31.648725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.648691 4471 generic.go:356] "Generic (PLEG): container finished" podID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" containerID="46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa" exitCode=137 Oct 31 00:52:31.648942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.648743 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-407e80" Oct 31 00:52:31.648942 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.648791 4471 scope.go:122] "RemoveContainer" containerID="46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa" Oct 31 00:52:31.658070 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.658048 4471 scope.go:122] "RemoveContainer" containerID="46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa" Oct 31 00:52:31.658377 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:31.658357 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa\": container with ID starting with 46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa not found: ID does not exist" containerID="46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa" Oct 31 00:52:31.658436 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:31.658387 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa"} err="failed to get container status \"46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa\": rpc error: code = NotFound desc = could not find container \"46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa\": container with ID starting with 46ef4b2661c192a442ae969a8bae226be8a51729effb39c3eba70ebce358d4aa not found: ID does not exist" Oct 31 00:52:33.128551 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:33.128500 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:34.280754 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:34.280717 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:52:34.711865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:34.711829 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:34.913329 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:34.913298 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:35.128894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.128807 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 00:52:35.129463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.129440 4471 cpu_manager.go:414] "containerMap: removing container" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" containerName="test-curl-7372-407e80" Oct 31 00:52:35.129463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.129463 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" containerName="test-curl-7372-407e80" Oct 31 00:52:35.129678 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.129654 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1c119a-8bf5-4034-baab-d3526c0fdf73" containerName="test-curl-7372-407e80" Oct 31 00:52:35.134707 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.134688 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.145122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.145094 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:35.187366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.187328 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bc2453-dockercfg-6s98j\"" Oct 31 00:52:35.187366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.187339 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bc2453\"" Oct 31 00:52:35.188308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.188280 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 00:52:35.205715 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.205677 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.205923 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.205849 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzcn7\" (UniqueName: \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.282299 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.282250 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:52:35.345553 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.322120 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.347302 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.347272 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzcn7\" (UniqueName: \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.347633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.324158 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.350090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.350069 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.350354 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.350330 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:52:35.408594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.408561 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzcn7\" (UniqueName: \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\") pod \"payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.448594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.448509 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8l2\" (UniqueName: \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.448771 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.448710 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:35.448850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.448796 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.550209 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.550171 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8l2\" (UniqueName: \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.550424 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.550401 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.551327 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.551298 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.568048 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.568012 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:35.611612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.611577 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8l2\" (UniqueName: \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\") pod \"payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.675445 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.675364 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:35.819829 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:52:35.819798 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50929a9c_8263_4806_a654_749734e3aa36.slice/crio-6b52de6cfc52387db57fa7c1e2200d743c67fe5c8572d74fa3e94839da164326 WatchSource:0}: Error finding container 6b52de6cfc52387db57fa7c1e2200d743c67fe5c8572d74fa3e94839da164326: Status 404 returned error can't find the container with id 6b52de6cfc52387db57fa7c1e2200d743c67fe5c8572d74fa3e94839da164326 Oct 31 00:52:35.822246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:35.822221 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 00:52:36.047153 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:52:36.047121 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd92850_978d_4298_8fb0_598f40ae70b0.slice/crio-6098d07564f3aa20fbe362803012dbbe51bf41a9813791b35aef4ef892f19dca WatchSource:0}: Error finding container 6098d07564f3aa20fbe362803012dbbe51bf41a9813791b35aef4ef892f19dca: Status 404 returned error can't find the container with id 6098d07564f3aa20fbe362803012dbbe51bf41a9813791b35aef4ef892f19dca Oct 31 00:52:36.049635 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.049613 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:52:36.560794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.560734 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:36.680918 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.680883 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" event={"ID":"50929a9c-8263-4806-a654-749734e3aa36","Type":"ContainerStarted","Data":"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2"} Oct 31 00:52:36.681130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.680940 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" event={"ID":"50929a9c-8263-4806-a654-749734e3aa36","Type":"ContainerStarted","Data":"6b52de6cfc52387db57fa7c1e2200d743c67fe5c8572d74fa3e94839da164326"} Oct 31 00:52:36.681130 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.681079 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:36.682419 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.682393 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" event={"ID":"fbd92850-978d-4298-8fb0-598f40ae70b0","Type":"ContainerStarted","Data":"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895"} Oct 31 00:52:36.682624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.682434 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" event={"ID":"fbd92850-978d-4298-8fb0-598f40ae70b0","Type":"ContainerStarted","Data":"6098d07564f3aa20fbe362803012dbbe51bf41a9813791b35aef4ef892f19dca"} Oct 31 00:52:36.682624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.682474 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:36.708006 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.707976 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:36.799633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.799574 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" podStartSLOduration=1.7995585840000001 podStartE2EDuration="1.799558584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:36.796762079 +0000 UTC m=+2283.722031032" watchObservedRunningTime="2026-10-31 00:52:36.799558584 +0000 UTC m=+2283.724827539" Oct 31 00:52:36.906336 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.906302 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:52:36.906587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.906557 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" containerID="cri-o://9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99" gracePeriod=30 Oct 31 00:52:36.915212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.915162 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" podStartSLOduration=1.915148456 podStartE2EDuration="1.915148456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:36.913923511 +0000 UTC m=+2283.839192464" watchObservedRunningTime="2026-10-31 00:52:36.915148456 +0000 UTC m=+2283.840417446" Oct 31 00:52:36.943216 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:36.943165 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:37.950612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:37.950577 4471 patch_prober.go:29] interesting pod/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:52:37.950994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:37.950640 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:52:40.282207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:40.282168 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.193244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.192933 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:52:42.246867 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.246834 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.246947 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.246986 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247046 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247073 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247089 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247143 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247145 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.247280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247205 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247246 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\") pod \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\" (UID: \"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5\") " Oct 31 00:52:42.247509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247394 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.247509 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247411 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.247670 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247600 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.247935 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.247910 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:42.248036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.248012 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.248101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.248044 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.248101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.248060 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.248101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.248073 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.249513 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.249484 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:42.249631 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.249566 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "kube-api-access-xjz9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:42.249736 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.249717 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:42.249795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.249729 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy" pod "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" (UID: "d2fd9edd-18e7-4e0e-89ce-c8570fd907c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:42.336774 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.336741 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.349395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.349319 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.349395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.349352 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.349395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.349365 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.349395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.349379 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.349395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.349393 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjz9h\" (UniqueName: \"kubernetes.io/projected/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5-kube-api-access-xjz9h\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:42.502351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.502254 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:42.717545 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.717501 4471 generic.go:356] "Generic (PLEG): container finished" podID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerID="9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99" exitCode=0 Oct 31 00:52:42.717738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.717565 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" Oct 31 00:52:42.717738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.717598 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" event={"ID":"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5","Type":"ContainerDied","Data":"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99"} Oct 31 00:52:42.717738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.717657 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p" event={"ID":"d2fd9edd-18e7-4e0e-89ce-c8570fd907c5","Type":"ContainerDied","Data":"d25d8f4cb5280ed034cd26e6632fa9a1764ec93962b55a3c646537ae0e5befcb"} Oct 31 00:52:42.717738 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.717686 4471 scope.go:122] "RemoveContainer" containerID="9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99" Oct 31 00:52:42.727850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.727659 4471 scope.go:122] "RemoveContainer" containerID="9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99" Oct 31 00:52:42.728086 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:52:42.728068 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99\": container with ID starting with 9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99 not found: ID does not exist" containerID="9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99" Oct 31 00:52:42.728138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.728098 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99"} err="failed to get container status \"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99\": rpc error: code = NotFound desc = could not find container \"9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99\": container with ID starting with 9423250844d98a38c5c3758ce944fc4b9d2c1d46e767aa880366fe22a0defa99 not found: ID does not exist" Oct 31 00:52:42.884676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.884591 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:52:42.935328 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:42.935295 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-982ae3-openshift-default-64d9fb649f-vb78p"] Oct 31 00:52:43.214603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:43.214563 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" path="/var/lib/kubelet/pods/d2fd9edd-18e7-4e0e-89ce-c8570fd907c5/volumes" Oct 31 00:52:45.480558 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:45.480510 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:47.691455 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:47.691424 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:52:47.691961 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:47.691493 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 00:52:50.484355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:50.484320 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:52.925813 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:52.925778 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:55.496311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:55.496278 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:59.067487 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.067451 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:59.534917 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.534882 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 00:52:59.535463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.535448 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" Oct 31 00:52:59.535512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.535465 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" Oct 31 00:52:59.535588 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.535577 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fd9edd-18e7-4e0e-89ce-c8570fd907c5" containerName="istio-proxy" Oct 31 00:52:59.538685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.538668 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.584495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.584463 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 00:52:59.588400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.588379 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621084 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wxg\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621122 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621123 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621150 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621280 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621316 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621367 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621406 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621427 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621491 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621516 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621602 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621623 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621655 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621650 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2vs\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621697 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621739 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.621823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621756 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621802 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.621894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.621832 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.677171 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.677137 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 00:52:59.722936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.722889 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.722971 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wxg\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723008 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723032 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.723121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723092 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723390 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723129 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.723518 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723497 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723671 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723171 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.723760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723707 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.723760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723739 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723790 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723865 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723822 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723872 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.723969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.723969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723939 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.724113 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.723989 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2vs\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.724113 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.724069 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.724212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.724117 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.724212 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.724145 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.724312 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.724232 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.724891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.724858 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.725394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.725361 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.725493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.725476 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.725806 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.725784 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.725883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.725866 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.726365 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.726031 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.726887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.726655 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.726887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.726841 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.727341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.727316 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.728730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.728693 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.729615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.729595 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.732118 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.732095 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.750898 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.750867 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 00:52:59.786015 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.785927 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:52:59.790383 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.790184 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2vs\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.790706 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.790688 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wxg\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.791461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.791441 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-cmr8q\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.792082 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.792057 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6m7ds\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:52:59.851697 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.851655 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:52:59.903121 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:52:59.902825 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:53:00.326347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.326309 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 00:53:00.358939 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:53:00.358911 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa585d6_b179_479c_96d9_bc4c74fbcf33.slice/crio-ca40842f17d47e3af107ce5967e7d34f1fd092482beef074f698851e5d8bd8e3 WatchSource:0}: Error finding container ca40842f17d47e3af107ce5967e7d34f1fd092482beef074f698851e5d8bd8e3: Status 404 returned error can't find the container with id ca40842f17d47e3af107ce5967e7d34f1fd092482beef074f698851e5d8bd8e3 Oct 31 00:53:00.361394 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.361356 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.361493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.361428 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.361493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.361460 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.375702 ip-10-0-134-11 kubenswrapper[4471]: W1031 00:53:00.375672 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef373e9_98e7_4e70_a65c_d95a0f84001c.slice/crio-f66b207a8298ea11cb5dc2bfa7fb5e92615f5e80610bc76d7f52e67629f52e90 WatchSource:0}: Error finding container f66b207a8298ea11cb5dc2bfa7fb5e92615f5e80610bc76d7f52e67629f52e90: Status 404 returned error can't find the container with id f66b207a8298ea11cb5dc2bfa7fb5e92615f5e80610bc76d7f52e67629f52e90 Oct 31 00:53:00.377541 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.377506 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 00:53:00.378815 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.378784 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.378904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.378845 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.378904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.378884 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:53:00.822289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.822248 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" event={"ID":"9ef373e9-98e7-4e70-a65c-d95a0f84001c","Type":"ContainerStarted","Data":"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221"} Oct 31 00:53:00.822493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.822311 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" event={"ID":"9ef373e9-98e7-4e70-a65c-d95a0f84001c","Type":"ContainerStarted","Data":"f66b207a8298ea11cb5dc2bfa7fb5e92615f5e80610bc76d7f52e67629f52e90"} Oct 31 00:53:00.824078 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.824044 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" event={"ID":"dfa585d6-b179-479c-96d9-bc4c74fbcf33","Type":"ContainerStarted","Data":"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32"} Oct 31 00:53:00.824195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.824091 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" event={"ID":"dfa585d6-b179-479c-96d9-bc4c74fbcf33","Type":"ContainerStarted","Data":"ca40842f17d47e3af107ce5967e7d34f1fd092482beef074f698851e5d8bd8e3"} Oct 31 00:53:00.852486 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.852449 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:53:00.903704 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.903670 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:53:00.942270 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:00.942219 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podStartSLOduration=1.942204314 podStartE2EDuration="1.942204314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:00.94193967 +0000 UTC m=+2307.867208623" watchObservedRunningTime="2026-10-31 00:53:00.942204314 +0000 UTC m=+2307.867473268" Oct 31 00:53:01.092641 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.088937 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podStartSLOduration=2.088914597 podStartE2EDuration="2.088914597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:01.068733258 +0000 UTC m=+2307.994002211" watchObservedRunningTime="2026-10-31 00:53:01.088914597 +0000 UTC m=+2308.014183550" Oct 31 00:53:01.410554 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.410003 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:01.647303 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.647269 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:01.809608 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.809276 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:01.853427 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.853391 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:01.853610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.853474 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:01.911055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.911021 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:01.911230 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:01.911089 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:02.383173 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:02.383080 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:02.852276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:02.852243 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:02.852709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:02.852300 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:02.906652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:02.905131 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:02.906652 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:02.905208 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:03.852042 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:03.852003 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:03.852223 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:03.852066 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:03.908477 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:03.908437 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:03.909110 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:03.909082 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:04.329143 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:53:04.329106 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:53:04.852195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:04.852161 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:04.852457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:04.852434 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:04.911593 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:04.911557 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:04.912098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:04.912072 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:05.852144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:05.852112 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:05.852382 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:05.852361 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:05.912725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:05.912683 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:05.913253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:05.912747 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:06.852719 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:06.852680 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:06.852900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:06.852758 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:06.904083 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:06.904045 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:06.904289 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:06.904127 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:06.925550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:06.925503 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:07.770188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.770155 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:07.853000 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.852959 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cmr8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 31 00:53:07.853161 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.853033 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 31 00:53:07.904767 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.904727 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6m7ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:53:07.904945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.904798 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 31 00:53:07.923967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:07.923930 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:08.840521 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:08.840479 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:08.985092 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:08.985058 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:53:08.985921 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:08.985895 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:53:08.986924 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:08.986898 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 00:53:08.993968 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:08.993943 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:09.070116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.070076 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:53:09.071112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.071084 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:53:09.071863 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.071838 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 00:53:09.198883 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.198834 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:09.212576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.212542 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:53:09.212793 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:09.212768 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" containerID="cri-o://c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e" gracePeriod=30 Oct 31 00:53:11.575153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:11.575111 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:13.180456 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:13.180417 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:13.467893 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:13.467797 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:13.622184 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:13.622143 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:14.147712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.147676 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:14.471023 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.470992 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:53:14.562071 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562033 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562148 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562288 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562226 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562306 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562365 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562416 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562411 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562449 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.562623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562546 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562593 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562613 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.562831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562646 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:14.562831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562666 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\") pod \"4bdac5ef-9fbe-4685-b550-04132790323c\" (UID: \"4bdac5ef-9fbe-4685-b550-04132790323c\") " Oct 31 00:53:14.562936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.562922 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.563112 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.563092 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.563357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.563272 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.563357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.563294 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.563357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.563309 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdac5ef-9fbe-4685-b550-04132790323c-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.563357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.563325 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.565100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.565076 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:14.565244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.565219 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "kube-api-access-ck4jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:14.565244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.565234 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:14.565435 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.565419 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token" pod "4bdac5ef-9fbe-4685-b550-04132790323c" (UID: "4bdac5ef-9fbe-4685-b550-04132790323c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:14.665177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.665136 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.665177 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.665170 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdac5ef-9fbe-4685-b550-04132790323c-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.665391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.665186 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck4jv\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-kube-api-access-ck4jv\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.665391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.665201 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdac5ef-9fbe-4685-b550-04132790323c-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.665391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.665212 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdac5ef-9fbe-4685-b550-04132790323c-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:14.925710 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.925675 4471 generic.go:356] "Generic (PLEG): container finished" podID="4bdac5ef-9fbe-4685-b550-04132790323c" containerID="c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e" exitCode=0 Oct 31 00:53:14.925890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.925725 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" Oct 31 00:53:14.925890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.925764 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" event={"ID":"4bdac5ef-9fbe-4685-b550-04132790323c","Type":"ContainerDied","Data":"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e"} Oct 31 00:53:14.925890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.925830 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f" event={"ID":"4bdac5ef-9fbe-4685-b550-04132790323c","Type":"ContainerDied","Data":"fe7606d7352f4d29ab1de7809c45e927cc80f319d07dc01b8d0faf9d15666cfa"} Oct 31 00:53:14.925890 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.925861 4471 scope.go:122] "RemoveContainer" containerID="c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e" Oct 31 00:53:14.936562 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.936521 4471 scope.go:122] "RemoveContainer" containerID="c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e" Oct 31 00:53:14.936864 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:53:14.936840 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e\": container with ID starting with c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e not found: ID does not exist" containerID="c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e" Oct 31 00:53:14.936926 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:14.936876 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e"} err="failed to get container status \"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e\": rpc error: code = NotFound desc = could not find container \"c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e\": container with ID starting with c748a4e67053ea4e9aab3948ddb2de04946e0295d7c070f3cfd2b7be21ee155e not found: ID does not exist" Oct 31 00:53:15.091418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:15.091377 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:53:15.141085 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:15.141040 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5b4db8-openshift-default-5cc6998bbc-gvj7f"] Oct 31 00:53:15.210126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:15.210052 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" path="/var/lib/kubelet/pods/4bdac5ef-9fbe-4685-b550-04132790323c/volumes" Oct 31 00:53:19.063580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:19.063545 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:19.207523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:19.207478 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:21.871001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:21.870965 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:22.015690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:22.015632 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.444975 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:24.444934 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.609182 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:24.609131 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:24.761028 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:24.760929 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:29.087870 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:29.087835 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:29.248237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:29.248195 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:29.998201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:29.998164 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:34.280871 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:53:34.280837 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:53:35.562974 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:35.562938 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:39.868852 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:39.868806 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:41.468444 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:41.468405 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:41.759685 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:41.759595 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:41.912450 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:41.912392 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:42.449594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:42.449562 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:46.977726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:46.977695 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:48.829374 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:48.829335 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:48.986377 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:48.986339 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:49.207242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:49.207185 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:49.212933 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:49.212903 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:53:49.213195 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:49.213163 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" containerID="cri-o://bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152" gracePeriod=30 Oct 31 00:53:51.608231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:51.608196 4471 patch_prober.go:29] interesting pod/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:53:51.608658 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:51.608256 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:53:51.914654 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:51.914602 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:54.256362 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:53:54.256329 4471 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e3de49_d2d9_4f71_be1a_de0effb425c5.slice/crio-bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:53:54.488351 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.488294 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:53:54.638734 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638696 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.638937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638763 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.638937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638791 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.638937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638838 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.638937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638895 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.639154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638950 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.639154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.638984 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.639154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639030 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.639154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639050 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.639154 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639054 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.639395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639252 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\") pod \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\" (UID: \"a6e3de49-d2d9-4f71-be1a-de0effb425c5\") " Oct 31 00:53:54.639395 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639285 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.639515 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639488 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.639613 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639560 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:54.639843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639824 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.639843 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639843 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.639995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639852 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.639995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639860 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.639995 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.639871 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.641257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.641227 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:53:54.641355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.641264 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:54.641463 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.641442 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:53:54.641523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.641502 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv" pod "a6e3de49-d2d9-4f71-be1a-de0effb425c5" (UID: "a6e3de49-d2d9-4f71-be1a-de0effb425c5"). InnerVolumeSpecName "kube-api-access-w6fsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:54.740839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.740755 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.740839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.740787 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.740839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.740796 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3de49-d2d9-4f71-be1a-de0effb425c5-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:54.740839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:54.740806 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6fsv\" (UniqueName: \"kubernetes.io/projected/a6e3de49-d2d9-4f71-be1a-de0effb425c5-kube-api-access-w6fsv\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:55.213881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.213845 4471 generic.go:356] "Generic (PLEG): container finished" podID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerID="bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152" exitCode=0 Oct 31 00:53:55.214034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.213909 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" Oct 31 00:53:55.214034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.213919 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" event={"ID":"a6e3de49-d2d9-4f71-be1a-de0effb425c5","Type":"ContainerDied","Data":"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152"} Oct 31 00:53:55.214034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.213979 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf" event={"ID":"a6e3de49-d2d9-4f71-be1a-de0effb425c5","Type":"ContainerDied","Data":"64963f7ea5b5a41eaf6fc74ae2fbd92c51312c9594f2422a7fe527d56147f7ee"} Oct 31 00:53:55.214034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.213993 4471 scope.go:122] "RemoveContainer" containerID="bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152" Oct 31 00:53:55.224487 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.224466 4471 scope.go:122] "RemoveContainer" containerID="bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152" Oct 31 00:53:55.224825 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:53:55.224801 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152\": container with ID starting with bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152 not found: ID does not exist" containerID="bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152" Oct 31 00:53:55.224940 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.224838 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152"} err="failed to get container status \"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152\": rpc error: code = NotFound desc = could not find container \"bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152\": container with ID starting with bb6cb69b04fb4f70f67c50a2ef602c97cca12e4a4e3bcb32728d6a993b096152 not found: ID does not exist" Oct 31 00:53:55.253580 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.253547 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:55.383996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.383958 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:53:55.387640 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:55.387606 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e8ba06-openshift-default-654dcd546f-qwvpf"] Oct 31 00:53:56.569410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:56.569366 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:56.886392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:56.886303 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:53:57.209523 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:53:57.209491 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" path="/var/lib/kubelet/pods/a6e3de49-d2d9-4f71-be1a-de0effb425c5/volumes" Oct 31 00:54:01.220115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:01.220063 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.471724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.471692 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.619345 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.619309 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.809279 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.809124 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:02.835431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.835399 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:54:02.835654 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.835630 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" containerID="cri-o://4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85" gracePeriod=30 Oct 31 00:54:02.857673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.857640 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:54:02.857879 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.857857 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" containerID="cri-o://343658fd226d56679218b45c50a0698a9852842e7077fa60cdb8483f494b40ec" gracePeriod=30 Oct 31 00:54:02.870692 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.870638 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:54:02.870887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.870849 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" containerID="cri-o://8c831e87ec2b335a835b1082aaf08582318a8e83371f3fb27cd3b4e3f82a16ff" gracePeriod=30 Oct 31 00:54:02.885389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.885351 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:54:02.885616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.885589 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" containerID="cri-o://e83938c08b42093461ed4ecdcc98609660d8b83f633b95cc5e84f0683fc2a124" gracePeriod=30 Oct 31 00:54:02.978900 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:02.978839 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:03.128370 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:03.128290 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:03.776016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:03.775984 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:04.284685 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:04.284651 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:54:04.543244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:04.543156 4471 patch_prober.go:29] interesting pod/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:54:04.543244 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:04.543233 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:54:08.196627 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.196160 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:54:08.321722 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.321689 4471 generic.go:356] "Generic (PLEG): container finished" podID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerID="e83938c08b42093461ed4ecdcc98609660d8b83f633b95cc5e84f0683fc2a124" exitCode=0 Oct 31 00:54:08.321904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.321866 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" event={"ID":"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8","Type":"ContainerDied","Data":"e83938c08b42093461ed4ecdcc98609660d8b83f633b95cc5e84f0683fc2a124"} Oct 31 00:54:08.323404 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.323374 4471 generic.go:356] "Generic (PLEG): container finished" podID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerID="8c831e87ec2b335a835b1082aaf08582318a8e83371f3fb27cd3b4e3f82a16ff" exitCode=0 Oct 31 00:54:08.323511 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.323469 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" event={"ID":"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4","Type":"ContainerDied","Data":"8c831e87ec2b335a835b1082aaf08582318a8e83371f3fb27cd3b4e3f82a16ff"} Oct 31 00:54:08.324664 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.324644 4471 generic.go:356] "Generic (PLEG): container finished" podID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerID="4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85" exitCode=0 Oct 31 00:54:08.324773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.324714 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" event={"ID":"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6","Type":"ContainerDied","Data":"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85"} Oct 31 00:54:08.324773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.324744 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" event={"ID":"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6","Type":"ContainerDied","Data":"4540569ec6bb1988598bd5270e63cb10f476e35bb306e2dba638a5f21a174149"} Oct 31 00:54:08.324773 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.324764 4471 scope.go:122] "RemoveContainer" containerID="4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85" Oct 31 00:54:08.324949 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.324886 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt" Oct 31 00:54:08.329001 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.328975 4471 generic.go:356] "Generic (PLEG): container finished" podID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerID="343658fd226d56679218b45c50a0698a9852842e7077fa60cdb8483f494b40ec" exitCode=0 Oct 31 00:54:08.329111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.329015 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerDied","Data":"343658fd226d56679218b45c50a0698a9852842e7077fa60cdb8483f494b40ec"} Oct 31 00:54:08.334720 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.334692 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.334831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.334779 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335016 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.334990 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335085 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335160 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335132 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335194 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335214 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335199 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.335358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335335 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335486 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335468 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335576 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335501 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\") pod \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\" (UID: \"a7fbdefc-9c49-42e5-b842-2ac8bd269ba6\") " Oct 31 00:54:08.335894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335413 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.336231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.336200 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.336231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.336221 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.336589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.336565 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.336668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335499 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.336668 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.335627 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.336798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.336766 4471 scope.go:122] "RemoveContainer" containerID="4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85" Oct 31 00:54:08.337112 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:08.337085 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85\": container with ID starting with 4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85 not found: ID does not exist" containerID="4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85" Oct 31 00:54:08.337186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.337138 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85"} err="failed to get container status \"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85\": rpc error: code = NotFound desc = could not find container \"4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85\": container with ID starting with 4ef4741fb4a1c8dfd40698c87c70167cde34be0f9923a27ed8356317b5f79b85 not found: ID does not exist" Oct 31 00:54:08.337186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.337165 4471 scope.go:122] "RemoveContainer" containerID="f1b58f2b6105866ec3bd2c124610f61131997d742f96f53ef033b8e3eb80cedf" Oct 31 00:54:08.337293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.337190 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.337864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.337839 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.339366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.339340 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.342236 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.342207 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm" pod "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" (UID: "a7fbdefc-9c49-42e5-b842-2ac8bd269ba6"). InnerVolumeSpecName "kube-api-access-fc5wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.342696 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.342674 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:54:08.349945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.349926 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:54:08.353689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.353667 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:54:08.390827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.390788 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:08.436939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436860 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.436939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436902 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.436939 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436927 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436967 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.436993 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437025 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437050 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437074 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437116 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437140 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437170 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437225 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437209 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437240 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437300 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437334 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437369 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437372 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437412 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437428 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437462 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437508 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437556 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437606 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437682 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437681 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437705 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.437741 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437739 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.438358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437905 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.438358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.437974 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.438358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438078 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.438358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438267 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\") pod \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\" (UID: \"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4\") " Oct 31 00:54:08.438358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438284 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.438510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438433 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.438510 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438474 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.438616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438515 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\") pod \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\" (UID: \"e47373da-96ae-46fc-be7e-ae1f5f014d7e\") " Oct 31 00:54:08.438616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438559 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\") pod \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\" (UID: \"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8\") " Oct 31 00:54:08.438954 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438478 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439203 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439231 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439247 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439261 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439275 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439290 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439303 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439317 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439331 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439347 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc5wm\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-kube-api-access-fc5wm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439365 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439383 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439397 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439396 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439410 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439451 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439468 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.439555 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439483 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.440712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439571 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.440712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.439700 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:08.440712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.438707 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.440712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.440651 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.441208 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.441184 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.441798 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.441711 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "kube-api-access-92wzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.441945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.441820 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.442859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.442831 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.442986 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.442967 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.443060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.442970 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.443060 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.443030 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token" pod "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" (UID: "f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.443248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.443228 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.443651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.443626 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:54:08.443884 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.443859 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "kube-api-access-j62lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.444004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.443987 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw" pod "e47373da-96ae-46fc-be7e-ae1f5f014d7e" (UID: "e47373da-96ae-46fc-be7e-ae1f5f014d7e"). InnerVolumeSpecName "kube-api-access-rflpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:08.444822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.444803 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy" pod "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" (UID: "6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:08.540814 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540776 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540834 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540850 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540859 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540871 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540883 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92wzb\" (UniqueName: \"kubernetes.io/projected/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-kube-api-access-92wzb\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540894 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540916 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540927 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540935 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j62lx\" (UniqueName: \"kubernetes.io/projected/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-kube-api-access-j62lx\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540944 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540953 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflpw\" (UniqueName: \"kubernetes.io/projected/e47373da-96ae-46fc-be7e-ae1f5f014d7e-kube-api-access-rflpw\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540962 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540970 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e47373da-96ae-46fc-be7e-ae1f5f014d7e-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540978 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.541005 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.540991 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:08.542224 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.542196 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:08.695785 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.695608 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:08.938494 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.938459 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:54:08.942837 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:08.942811 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-jztgt"] Oct 31 00:54:09.208904 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.208871 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" path="/var/lib/kubelet/pods/a7fbdefc-9c49-42e5-b842-2ac8bd269ba6/volumes" Oct 31 00:54:09.335742 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.335710 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" Oct 31 00:54:09.335957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.335706 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp" event={"ID":"6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4","Type":"ContainerDied","Data":"4589d98142e2d65851012612432998b81e70edf139e2e3a35562c2955eb370f9"} Oct 31 00:54:09.335957 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.335878 4471 scope.go:122] "RemoveContainer" containerID="8c831e87ec2b335a835b1082aaf08582318a8e83371f3fb27cd3b4e3f82a16ff" Oct 31 00:54:09.338514 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.338486 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" event={"ID":"e47373da-96ae-46fc-be7e-ae1f5f014d7e","Type":"ContainerDied","Data":"73fa4ebf19ec67331f1414e9d09728e94bf6e6be2cab8f455923dde180b88687"} Oct 31 00:54:09.338705 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.338611 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj" Oct 31 00:54:09.340105 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.340089 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" Oct 31 00:54:09.340971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.340149 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm" event={"ID":"f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8","Type":"ContainerDied","Data":"e0784ceaf1efc0fab72d585286d335589fe7357e44aad8a7f1d77427891ab314"} Oct 31 00:54:09.349321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.349292 4471 scope.go:122] "RemoveContainer" containerID="343658fd226d56679218b45c50a0698a9852842e7077fa60cdb8483f494b40ec" Oct 31 00:54:09.380565 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.376723 4471 scope.go:122] "RemoveContainer" containerID="e83938c08b42093461ed4ecdcc98609660d8b83f633b95cc5e84f0683fc2a124" Oct 31 00:54:09.584123 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.584019 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:54:09.586577 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.586549 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-f9ghj"] Oct 31 00:54:09.689566 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.689502 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:54:09.693985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.693954 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-fcctm"] Oct 31 00:54:09.803572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.803522 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:54:09.809805 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:09.809772 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d5f4de7-openshift-default-55c88dfdbf-6cwlp"] Oct 31 00:54:10.237731 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:10.237692 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:10.385950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:10.385909 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:10.583614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:10.583310 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:11.122197 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:11.122158 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:11.209066 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:11.209034 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" path="/var/lib/kubelet/pods/6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4/volumes" Oct 31 00:54:11.209558 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:11.209542 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" path="/var/lib/kubelet/pods/e47373da-96ae-46fc-be7e-ae1f5f014d7e/volumes" Oct 31 00:54:11.210007 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:11.209990 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" path="/var/lib/kubelet/pods/f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8/volumes" Oct 31 00:54:13.719069 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:13.719030 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:14.544229 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:14.544191 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:16.081498 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.081462 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:54:16.210780 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.210746 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:16.410149 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.410117 4471 generic.go:356] "Generic (PLEG): container finished" podID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerID="a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863" exitCode=0 Oct 31 00:54:16.410321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.410185 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerDied","Data":"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863"} Oct 31 00:54:16.410616 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.410598 4471 scope.go:122] "RemoveContainer" containerID="a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863" Oct 31 00:54:16.411990 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.411964 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:54:16.412089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.412040 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:54:16.412089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:16.412075 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 00:54:17.418231 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:17.418191 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerStarted","Data":"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757"} Oct 31 00:54:18.037401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:18.037368 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:54:18.037614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:18.037551 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:54:18.037614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:18.037601 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:54:19.037482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:19.037437 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:54:19.038036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:19.037511 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:54:19.113546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:19.113499 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:20.037901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:20.037860 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:54:20.038341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:20.037929 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:54:21.037518 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:21.037478 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:54:21.037714 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:21.037555 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:54:22.037495 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:22.037456 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:54:22.037998 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:22.037510 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:54:23.042457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.042412 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:54:23.042971 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.042725 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:54:23.043610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.043592 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:54:23.073393 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.073366 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:23.264676 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.264641 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:54:23.479959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.479922 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:23.624387 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:23.624350 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:24.312830 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:24.312787 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:26.441366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:26.441330 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:28.045441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:28.045353 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:54:28.193493 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:28.193456 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:28.265612 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:28.265562 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" containerID="cri-o://736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02" gracePeriod=25 Oct 31 00:54:29.596689 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:29.596651 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:29.820673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:29.820643 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:31.104417 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.104365 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerName="main" containerID="cri-o://845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04" gracePeriod=45 Oct 31 00:54:31.366575 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.366510 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:54:31.453731 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453693 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\" (UniqueName: \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.453937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453838 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.453937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453857 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.453937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453889 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.453937 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453918 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.454159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.453957 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\") pod \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\" (UID: \"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c\") " Oct 31 00:54:31.454159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454086 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.454159 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454134 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.454311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454193 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.454446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454427 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.454512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454446 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.454512 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.454456 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.456162 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.456114 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:31.456269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.456175 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "kube-api-access-q5qtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:31.456269 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.456208 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs" pod "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" (UID: "ed5849bf-a0ae-46a0-bc80-adbe6bedc03c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:31.505180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.498591 4471 generic.go:356] "Generic (PLEG): container finished" podID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerID="845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04" exitCode=0 Oct 31 00:54:31.505180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.498642 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" event={"ID":"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c","Type":"ContainerDied","Data":"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04"} Oct 31 00:54:31.505180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.498676 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" event={"ID":"ed5849bf-a0ae-46a0-bc80-adbe6bedc03c","Type":"ContainerDied","Data":"cd085c1ace40bd1da67884e385e174e6cd86d400271d250077729e7186e9cbeb"} Oct 31 00:54:31.505180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.498696 4471 scope.go:122] "RemoveContainer" containerID="845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04" Oct 31 00:54:31.505180 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.498858 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2" Oct 31 00:54:31.513297 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.513272 4471 scope.go:122] "RemoveContainer" containerID="845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04" Oct 31 00:54:31.513638 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:31.513616 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04\": container with ID starting with 845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04 not found: ID does not exist" containerID="845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04" Oct 31 00:54:31.513726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.513645 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04"} err="failed to get container status \"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04\": rpc error: code = NotFound desc = could not find container \"845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04\": container with ID starting with 845e3dc9cf21a17ec60de6d87c4bc8ef8d429e0d4c45ae0b0be2302b1de4aa04 not found: ID does not exist" Oct 31 00:54:31.555418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.555379 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.555418 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.555418 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.555753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.555433 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qtc\" (UniqueName: \"kubernetes.io/projected/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c-kube-api-access-q5qtc\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:31.670019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.669985 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:54:31.677084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:31.677041 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-1e5e75/distinct-2-w3-1e5e75-kserve-654d49f9bf-6l4s2"] Oct 31 00:54:32.267017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:32.266970 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:33.228643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:33.228167 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" path="/var/lib/kubelet/pods/ed5849bf-a0ae-46a0-bc80-adbe6bedc03c/volumes" Oct 31 00:54:34.297245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:34.297073 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:54:34.297245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:34.297143 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:54:34.639270 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:34.639138 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:54:35.239039 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:35.239005 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:35.650698 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:35.650655 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:35.840040 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:35.840007 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:40.387307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:40.387271 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:42.267061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:42.267014 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:54:43.067238 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.067200 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerName="main" containerID="cri-o://d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315" gracePeriod=45 Oct 31 00:54:43.330248 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.330178 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:54:43.430958 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.430923 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.430969 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\" (UniqueName: \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431000 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\" (UniqueName: \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431030 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431112 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431191 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\") pod \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\" (UID: \"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3\") " Oct 31 00:54:43.431357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431245 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.431517 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431449 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.431683 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431579 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.431907 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431890 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.431979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431911 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.431979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.431948 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.433542 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.433507 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:54:43.433620 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.433547 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:54:43.433677 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.433658 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2" pod "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" (UID: "29b30eb3-3d46-43fa-8fcc-28bfb444ebe3"). InnerVolumeSpecName "kube-api-access-4zdh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:43.533570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.533509 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.533570 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.533571 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zdh2\" (UniqueName: \"kubernetes.io/projected/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-kube-api-access-4zdh2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.533768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.533588 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:43.658457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.658421 4471 generic.go:356] "Generic (PLEG): container finished" podID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerID="d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315" exitCode=0 Oct 31 00:54:43.658457 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.658459 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" Oct 31 00:54:43.658724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.658503 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" event={"ID":"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3","Type":"ContainerDied","Data":"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315"} Oct 31 00:54:43.658724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.658565 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj" event={"ID":"29b30eb3-3d46-43fa-8fcc-28bfb444ebe3","Type":"ContainerDied","Data":"a62decde01a0f4dbcc32481545c3c1793b9a1b2342ec56b938faa42f50906dda"} Oct 31 00:54:43.658724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.658585 4471 scope.go:122] "RemoveContainer" containerID="d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315" Oct 31 00:54:43.669175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.669154 4471 scope.go:122] "RemoveContainer" containerID="d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315" Oct 31 00:54:43.669520 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:43.669500 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315\": container with ID starting with d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315 not found: ID does not exist" containerID="d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315" Oct 31 00:54:43.669606 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.669544 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315"} err="failed to get container status \"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315\": rpc error: code = NotFound desc = could not find container \"d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315\": container with ID starting with d89352e2a6774dfd1d1395cc824117b328fa5d4caaabe840ef6060a4d0a5d315 not found: ID does not exist" Oct 31 00:54:43.823123 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.823080 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:54:43.826293 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:43.826269 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6f4a5/premium-w6-a6f4a5-kserve-7ff5fc86d5-hjdfj"] Oct 31 00:54:45.209090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:45.209056 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" path="/var/lib/kubelet/pods/29b30eb3-3d46-43fa-8fcc-28bfb444ebe3/volumes" Oct 31 00:54:45.382276 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:45.382231 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:46.955308 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:46.955229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:48.539189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.539130 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:54:48.688908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.688876 4471 generic.go:356] "Generic (PLEG): container finished" podID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerID="736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02" exitCode=0 Oct 31 00:54:48.689100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.688937 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" Oct 31 00:54:48.689100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.688959 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" event={"ID":"763d31cb-8503-4af7-a2b0-fe264ed5604b","Type":"ContainerDied","Data":"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02"} Oct 31 00:54:48.689100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.689012 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p" event={"ID":"763d31cb-8503-4af7-a2b0-fe264ed5604b","Type":"ContainerDied","Data":"e58d1e8a552813aa8d395fcaa9b702b579cd4fe7f84f5cdff037b4e4ed615546"} Oct 31 00:54:48.689100 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.689042 4471 scope.go:122] "RemoveContainer" containerID="736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02" Oct 31 00:54:48.698855 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.698835 4471 scope.go:122] "RemoveContainer" containerID="736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02" Oct 31 00:54:48.699117 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:54:48.699097 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02\": container with ID starting with 736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02 not found: ID does not exist" containerID="736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02" Oct 31 00:54:48.699193 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.699130 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02"} err="failed to get container status \"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02\": rpc error: code = NotFound desc = could not find container \"736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02\": container with ID starting with 736a4f8b69daf73eda0376fb281292b5aba512436215addd006f4f2c89389a02 not found: ID does not exist" Oct 31 00:54:48.702737 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.702716 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\") pod \"763d31cb-8503-4af7-a2b0-fe264ed5604b\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " Oct 31 00:54:48.702833 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.702818 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\" (UniqueName: \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\") pod \"763d31cb-8503-4af7-a2b0-fe264ed5604b\" (UID: \"763d31cb-8503-4af7-a2b0-fe264ed5604b\") " Oct 31 00:54:48.703115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.703088 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume" pod "763d31cb-8503-4af7-a2b0-fe264ed5604b" (UID: "763d31cb-8503-4af7-a2b0-fe264ed5604b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:54:48.703311 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.703292 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763d31cb-8503-4af7-a2b0-fe264ed5604b-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:48.705065 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.705039 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg" pod "763d31cb-8503-4af7-a2b0-fe264ed5604b" (UID: "763d31cb-8503-4af7-a2b0-fe264ed5604b"). InnerVolumeSpecName "kube-api-access-q59bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:54:48.804358 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:48.804270 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59bg\" (UniqueName: \"kubernetes.io/projected/763d31cb-8503-4af7-a2b0-fe264ed5604b-kube-api-access-q59bg\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:54:49.109272 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:49.109190 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:54:49.158948 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:49.158914 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-1e5e75-7bf6cbcdcf-6b95p"] Oct 31 00:54:49.208341 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:49.208308 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" path="/var/lib/kubelet/pods/763d31cb-8503-4af7-a2b0-fe264ed5604b/volumes" Oct 31 00:54:50.474623 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:50.474585 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.389124 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:51.389089 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.582357 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:51.582302 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.728860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:51.728824 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:51.892542 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:51.892476 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.221264 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:52.221229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:52.473615 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:52.473515 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:55.491831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:55.491796 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.130752 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:59.130717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:54:59.321116 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:54:59.321082 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:00.779932 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:00.779888 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:04.282483 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:55:04.282438 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:55:05.862959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:05.862920 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.673724 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:08.673686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:08.845343 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:08.845308 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:09.219889 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:09.219850 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:09.370423 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:09.370387 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:10.783794 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:10.783760 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.460603 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:15.460564 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:15.777088 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:15.776974 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:20.803097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:20.803056 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:22.504474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:22.504439 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:23.653938 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:23.653888 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:23.797246 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:23.797203 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.111546 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:24.111450 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:24.253909 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:24.253864 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:25.473730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:25.473694 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.251879 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:27.251847 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.480120 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:27.480087 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:55:27.560791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:27.560701 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:27.708136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:27.708099 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:28.172594 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:28.172558 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:29.786479 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:29.786445 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:32.811799 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:32.811767 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.280737 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:55:34.280703 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:55:34.528875 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.528821 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.669604 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.669570 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.843648 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.843542 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:34.878754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.878720 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:55:34.878994 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.878969 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" containerID="cri-o://4d244dd4b98699a46a3f91f8ca1e537fa50ba72f4a095e1855a556df5a5f8bed" gracePeriod=30 Oct 31 00:55:34.984881 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:34.984843 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:37.971768 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:37.971724 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:38.708236 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:38.708201 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:40.020008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.019977 4471 generic.go:356] "Generic (PLEG): container finished" podID="71de909a-5700-4360-8d6d-6551809d6a76" containerID="4d244dd4b98699a46a3f91f8ca1e537fa50ba72f4a095e1855a556df5a5f8bed" exitCode=0 Oct 31 00:55:40.020474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.020056 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerDied","Data":"4d244dd4b98699a46a3f91f8ca1e537fa50ba72f4a095e1855a556df5a5f8bed"} Oct 31 00:55:40.020474 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.020131 4471 scope.go:122] "RemoveContainer" containerID="bb5a6a1d87912dc734790d5aea16ad40700a828ac87fa7184230c931bb783a96" Oct 31 00:55:40.153739 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.153713 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:55:40.231696 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231658 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.231842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231808 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.231908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231857 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.231908 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231901 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231919 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.232010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231942 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.231980 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232006 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232042 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232124 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\") pod \"71de909a-5700-4360-8d6d-6551809d6a76\" (UID: \"71de909a-5700-4360-8d6d-6551809d6a76\") " Oct 31 00:55:40.232213 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232152 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.232391 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232360 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.232636 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232611 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.232730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232632 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.232730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232614 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:40.232730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232652 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.232730 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.232667 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.234497 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.234465 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:40.234656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.234514 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "kube-api-access-mfj9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:40.234656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.234582 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:40.234791 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.234777 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo" pod "71de909a-5700-4360-8d6d-6551809d6a76" (UID: "71de909a-5700-4360-8d6d-6551809d6a76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:40.334644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334603 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfj9z\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-kube-api-access-mfj9z\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.334644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334641 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71de909a-5700-4360-8d6d-6551809d6a76-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.334644 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334656 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.334901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334673 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71de909a-5700-4360-8d6d-6551809d6a76-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.334901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334686 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71de909a-5700-4360-8d6d-6551809d6a76-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:40.334901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:40.334700 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71de909a-5700-4360-8d6d-6551809d6a76-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:41.026801 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:41.026769 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" Oct 31 00:55:41.027307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:41.026801 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4" event={"ID":"71de909a-5700-4360-8d6d-6551809d6a76","Type":"ContainerDied","Data":"becf8894e4e31724b16a6865e8eb98fd351f33c354398e753a788810b76b51f0"} Oct 31 00:55:41.027307 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:41.026859 4471 scope.go:122] "RemoveContainer" containerID="4d244dd4b98699a46a3f91f8ca1e537fa50ba72f4a095e1855a556df5a5f8bed" Oct 31 00:55:41.241469 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:41.241430 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:55:41.291025 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:41.290947 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-1e5e75-openshift-default-879c67bc5-wx4r4"] Oct 31 00:55:42.500747 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.500709 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" podUID="5f11ac15-675a-4de5-be97-fe932e5fa986" containerName="main" containerID="cri-o://6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768" gracePeriod=45 Oct 31 00:55:42.769134 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.769102 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:55:42.864823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.864768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.864865 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\" (UniqueName: \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.864912 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865017 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.864973 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865074 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.865155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865097 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865155 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865151 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\") pod \"5f11ac15-675a-4de5-be97-fe932e5fa986\" (UID: \"5f11ac15-675a-4de5-be97-fe932e5fa986\") " Oct 31 00:55:42.865310 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865206 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.865408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865387 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.865633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865606 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.865633 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865632 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.865823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.865647 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.867198 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.867175 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:42.867306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.867209 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "kube-api-access-bbhbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:42.867306 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.867260 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm" pod "5f11ac15-675a-4de5-be97-fe932e5fa986" (UID: "5f11ac15-675a-4de5-be97-fe932e5fa986"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:42.967237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.967203 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11ac15-675a-4de5-be97-fe932e5fa986-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.967237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.967235 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhbp\" (UniqueName: \"kubernetes.io/projected/5f11ac15-675a-4de5-be97-fe932e5fa986-kube-api-access-bbhbp\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:42.967438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:42.967247 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11ac15-675a-4de5-be97-fe932e5fa986-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:43.042614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.042510 4471 generic.go:356] "Generic (PLEG): container finished" podID="5f11ac15-675a-4de5-be97-fe932e5fa986" containerID="6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768" exitCode=0 Oct 31 00:55:43.042761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.042667 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" event={"ID":"5f11ac15-675a-4de5-be97-fe932e5fa986","Type":"ContainerDied","Data":"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768"} Oct 31 00:55:43.042761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.042687 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" Oct 31 00:55:43.042761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.042707 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc" event={"ID":"5f11ac15-675a-4de5-be97-fe932e5fa986","Type":"ContainerDied","Data":"ca052fabe708146896555f4457c5bf1bb812430dcbb108a1c4e2e9dabb0a1bd6"} Oct 31 00:55:43.042761 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.042729 4471 scope.go:122] "RemoveContainer" containerID="6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768" Oct 31 00:55:43.063959 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.063932 4471 scope.go:122] "RemoveContainer" containerID="6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768" Oct 31 00:55:43.064312 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:55:43.064286 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768\": container with ID starting with 6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768 not found: ID does not exist" containerID="6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768" Oct 31 00:55:43.064422 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.064323 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768"} err="failed to get container status \"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768\": rpc error: code = NotFound desc = could not find container \"6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768\": container with ID starting with 6bb7ff5bbd93b2eb6d12aab50b778f4a301c6f0662ce01d6519a6e2259c9d768 not found: ID does not exist" Oct 31 00:55:43.209316 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.209279 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71de909a-5700-4360-8d6d-6551809d6a76" path="/var/lib/kubelet/pods/71de909a-5700-4360-8d6d-6551809d6a76/volumes" Oct 31 00:55:43.210098 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.210073 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:55:43.215979 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.215951 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-412743/premium-w5-412743-kserve-7c89d65946-ftpkc"] Oct 31 00:55:43.351218 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.351101 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:43.696333 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.696298 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:55:43.827392 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:43.827263 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:45.212291 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:45.212258 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f11ac15-675a-4de5-be97-fe932e5fa986" path="/var/lib/kubelet/pods/5f11ac15-675a-4de5-be97-fe932e5fa986/volumes" Oct 31 00:55:47.980401 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:47.980367 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:55:47.980911 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:47.980682 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" podUID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" containerName="maas-api" containerID="cri-o://2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf" gracePeriod=30 Oct 31 00:55:48.188168 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.188128 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:48.234753 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.234688 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:55:48.431748 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.431710 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\" (UniqueName: \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\") pod \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " Oct 31 00:55:48.431930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.431764 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") pod \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " Oct 31 00:55:48.431930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.431845 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\") pod \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " Oct 31 00:55:48.431930 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.431904 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\") pod \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\" (UID: \"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7\") " Oct 31 00:55:48.432389 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.432356 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca" pod "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" (UID: "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:48.434144 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.434100 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls" pod "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" (UID: "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:48.434247 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.434184 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96" pod "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" (UID: "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7"). InnerVolumeSpecName "kube-api-access-m9t96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:48.434399 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.434374 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs" pod "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" (UID: "8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:48.541136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.536945 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:48.541136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.536986 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:48.541136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.537001 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9t96\" (UniqueName: \"kubernetes.io/projected/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-kube-api-access-m9t96\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:48.541136 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:48.537018 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:49.080363 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.080330 4471 generic.go:356] "Generic (PLEG): container finished" podID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" containerID="2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf" exitCode=0 Oct 31 00:55:49.080844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.080387 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" Oct 31 00:55:49.080844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.080413 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" event={"ID":"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7","Type":"ContainerDied","Data":"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf"} Oct 31 00:55:49.080844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.080468 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn" event={"ID":"8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7","Type":"ContainerDied","Data":"0a136c84856c7b99808d2cfdd20ef008a96d7147e6465cee48e8138d8543d25a"} Oct 31 00:55:49.080844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.080490 4471 scope.go:122] "RemoveContainer" containerID="2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf" Oct 31 00:55:49.091454 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.091436 4471 scope.go:122] "RemoveContainer" containerID="2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf" Oct 31 00:55:49.091738 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:55:49.091708 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf\": container with ID starting with 2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf not found: ID does not exist" containerID="2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf" Oct 31 00:55:49.091795 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.091745 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf"} err="failed to get container status \"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf\": rpc error: code = NotFound desc = could not find container \"2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf\": container with ID starting with 2dd1fe7532536fae3302f7591df4f56f6caef464eeed97df8d22130e8bf2ecdf not found: ID does not exist" Oct 31 00:55:49.244444 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.244404 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:55:49.303874 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:49.296742 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a6f4a5-6744cdcf56-zd9bn"] Oct 31 00:55:50.097587 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:50.097487 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.245539 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:50.245488 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.393996 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:50.393962 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:50.655355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:50.655266 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:55:51.004887 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:51.004808 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.148765 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:51.148726 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:51.208680 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:51.208641 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" path="/var/lib/kubelet/pods/8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7/volumes" Oct 31 00:55:55.559036 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:55.558995 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:55.656034 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:55.655993 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" containerID="cri-o://d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398" gracePeriod=25 Oct 31 00:55:56.353823 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:56.353730 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:55:57.340145 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:57.340111 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:57.490431 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:57.490396 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:57.715733 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:57.710804 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:55:57.715733 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:57.711058 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" containerID="cri-o://a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420" gracePeriod=30 Oct 31 00:55:57.792439 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:57.792347 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:55:58.727858 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:58.727792 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerName="main" containerID="cri-o://fcd70532a19cd08cce74edb22fef9ba5b99916c77d8e93988ab3ce9a60597393" gracePeriod=45 Oct 31 00:55:58.947093 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:58.947027 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": dial tcp 10.133.0.41:8000: connect: connection refused" Oct 31 00:55:59.143458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.143421 4471 generic.go:356] "Generic (PLEG): container finished" podID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerID="fcd70532a19cd08cce74edb22fef9ba5b99916c77d8e93988ab3ce9a60597393" exitCode=0 Oct 31 00:55:59.143686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.143475 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" event={"ID":"6efb79f1-cdee-4336-936b-2157801f5e3a","Type":"ContainerDied","Data":"fcd70532a19cd08cce74edb22fef9ba5b99916c77d8e93988ab3ce9a60597393"} Oct 31 00:55:59.318901 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.315057 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:55:59.433860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433781 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\" (UniqueName: \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.433860 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433845 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.434101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433867 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.434101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433896 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.434101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433945 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.434101 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.433981 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\") pod \"6efb79f1-cdee-4336-936b-2157801f5e3a\" (UID: \"6efb79f1-cdee-4336-936b-2157801f5e3a\") " Oct 31 00:55:59.434319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434117 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.434319 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434178 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.434425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434385 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.434556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434506 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.434556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434550 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.434681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.434565 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.436153 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.436123 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:59.436257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.436179 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "kube-api-access-ksqg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:59.436257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.436206 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm" pod "6efb79f1-cdee-4336-936b-2157801f5e3a" (UID: "6efb79f1-cdee-4336-936b-2157801f5e3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:59.536364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.536322 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksqg9\" (UniqueName: \"kubernetes.io/projected/6efb79f1-cdee-4336-936b-2157801f5e3a-kube-api-access-ksqg9\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.536364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.536360 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efb79f1-cdee-4336-936b-2157801f5e3a-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.536364 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.536374 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efb79f1-cdee-4336-936b-2157801f5e3a-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:59.964268 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:55:59.964223 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:00.150842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.150816 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" Oct 31 00:56:00.151010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.150844 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc" event={"ID":"6efb79f1-cdee-4336-936b-2157801f5e3a","Type":"ContainerDied","Data":"9d4604643a96efa1aa6b7ebfa276916d87e992fad61b2bcb21e94feaea1be757"} Oct 31 00:56:00.151010 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.150900 4471 scope.go:122] "RemoveContainer" containerID="fcd70532a19cd08cce74edb22fef9ba5b99916c77d8e93988ab3ce9a60597393" Oct 31 00:56:00.318355 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.318269 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:56:00.367693 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.367659 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-101b9e/simulator-w0-101b9e-kserve-847db75867-tppvc"] Oct 31 00:56:00.994299 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:00.994268 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:01.207822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:01.207791 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" path="/var/lib/kubelet/pods/6efb79f1-cdee-4336-936b-2157801f5e3a/volumes" Oct 31 00:56:02.944912 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:02.944888 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:56:03.072896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.072810 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.072896 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.072873 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.072911 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.072948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073001 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073061 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073140 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073117 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.073408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073175 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073224 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073325 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\") pod \"a37f42c5-9658-4fdc-b34f-be9eec95403a\" (UID: \"a37f42c5-9658-4fdc-b34f-be9eec95403a\") " Oct 31 00:56:03.073408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073336 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.073408 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073398 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.073694 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073563 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.073765 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.074035 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.074060 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.074074 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37f42c5-9658-4fdc-b34f-be9eec95403a-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.074087 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.074125 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.074100 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.075251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.075226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.075366 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.075350 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "kube-api-access-24z8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:03.075483 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.075469 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:03.075585 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.075571 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy" pod "a37f42c5-9658-4fdc-b34f-be9eec95403a" (UID: "a37f42c5-9658-4fdc-b34f-be9eec95403a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:03.169044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.169006 4471 generic.go:356] "Generic (PLEG): container finished" podID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerID="a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420" exitCode=0 Oct 31 00:56:03.169207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.169061 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" Oct 31 00:56:03.169207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.169089 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" event={"ID":"a37f42c5-9658-4fdc-b34f-be9eec95403a","Type":"ContainerDied","Data":"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420"} Oct 31 00:56:03.169207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.169148 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5" event={"ID":"a37f42c5-9658-4fdc-b34f-be9eec95403a","Type":"ContainerDied","Data":"b16eb96829473c32d3a46cac4238d409f599eb88348ff6d6c196061179345152"} Oct 31 00:56:03.169207 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.169160 4471 scope.go:122] "RemoveContainer" containerID="a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420" Oct 31 00:56:03.174936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.174910 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.174936 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.174937 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.175089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.174953 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24z8s\" (UniqueName: \"kubernetes.io/projected/a37f42c5-9658-4fdc-b34f-be9eec95403a-kube-api-access-24z8s\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.175089 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.174966 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37f42c5-9658-4fdc-b34f-be9eec95403a-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:03.178764 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.178743 4471 scope.go:122] "RemoveContainer" containerID="a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420" Oct 31 00:56:03.179018 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:03.178996 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420\": container with ID starting with a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420 not found: ID does not exist" containerID="a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420" Oct 31 00:56:03.179073 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.179029 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420"} err="failed to get container status \"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420\": rpc error: code = NotFound desc = could not find container \"a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420\": container with ID starting with a1acc4d4f12338e7cec331f8bc228d1de523e1c7118ac141ce2b3fb438353420 not found: ID does not exist" Oct 31 00:56:03.334988 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.334906 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:56:03.339950 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.339921 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6f4a5-openshift-default-75d85b9b75-8prk5"] Oct 31 00:56:03.820760 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.820707 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:03.965019 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:03.964987 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:04.282896 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:04.282857 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:56:05.210138 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:05.210102 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" path="/var/lib/kubelet/pods/a37f42c5-9658-4fdc-b34f-be9eec95403a/volumes" Oct 31 00:56:05.399784 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:05.399743 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:05.981359 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:05.981314 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:06.353253 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:06.353162 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:56:11.236609 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:11.236577 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:11.382659 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:11.382614 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:11.533810 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:11.533719 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:12.957317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:12.957284 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:13.115447 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:13.115410 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:15.901149 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:15.901119 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:56:16.026283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.026204 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\") pod \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " Oct 31 00:56:16.026283 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.026248 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\" (UniqueName: \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\") pod \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\" (UID: \"aa043e5c-378c-43dd-997d-a3c7f0abcd72\") " Oct 31 00:56:16.026673 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.026651 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume" pod "aa043e5c-378c-43dd-997d-a3c7f0abcd72" (UID: "aa043e5c-378c-43dd-997d-a3c7f0abcd72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:16.026802 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.026781 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa043e5c-378c-43dd-997d-a3c7f0abcd72-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:16.028945 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.028916 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4" pod "aa043e5c-378c-43dd-997d-a3c7f0abcd72" (UID: "aa043e5c-378c-43dd-997d-a3c7f0abcd72"). InnerVolumeSpecName "kube-api-access-j7cb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:16.128174 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.128134 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7cb4\" (UniqueName: \"kubernetes.io/projected/aa043e5c-378c-43dd-997d-a3c7f0abcd72-kube-api-access-j7cb4\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:16.238287 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.238253 4471 generic.go:356] "Generic (PLEG): container finished" podID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerID="d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398" exitCode=0 Oct 31 00:56:16.238468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.238302 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" Oct 31 00:56:16.238468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.238321 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" event={"ID":"aa043e5c-378c-43dd-997d-a3c7f0abcd72","Type":"ContainerDied","Data":"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398"} Oct 31 00:56:16.238468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.238363 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7" event={"ID":"aa043e5c-378c-43dd-997d-a3c7f0abcd72","Type":"ContainerDied","Data":"950e99ab90e03861fa7758fd7041750415294f691d056f0a7b5f28de39df5c73"} Oct 31 00:56:16.238468 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.238383 4471 scope.go:122] "RemoveContainer" containerID="d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398" Oct 31 00:56:16.247390 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.247367 4471 scope.go:122] "RemoveContainer" containerID="d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398" Oct 31 00:56:16.247736 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:16.247717 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398\": container with ID starting with d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398 not found: ID does not exist" containerID="d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398" Oct 31 00:56:16.247827 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.247745 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398"} err="failed to get container status \"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398\": rpc error: code = NotFound desc = could not find container \"d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398\": container with ID starting with d0d8026cb68ca86e48c58f9a83b5992b6617cdd509c6d0ec4192db0a2e717398 not found: ID does not exist" Oct 31 00:56:16.402446 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.402412 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:56:16.408215 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.408192 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-101b9e-6c44b8947-88qn7"] Oct 31 00:56:16.473425 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:16.473387 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:17.208788 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:17.208757 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" path="/var/lib/kubelet/pods/aa043e5c-378c-43dd-997d-a3c7f0abcd72/volumes" Oct 31 00:56:21.323014 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:21.322970 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:21.467251 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:21.467213 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:21.614842 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:21.614760 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:22.253217 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:22.253181 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:26.474839 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:26.474798 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:28.218032 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:28.217966 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:30.139200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:30.139163 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:30.302346 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:30.302302 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.412702 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:31.412658 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:34.279568 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:34.279518 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:56:35.589210 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.589172 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:56:35.589686 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.589378 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" podUID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" containerName="maas-api" containerID="cri-o://5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35" gracePeriod=30 Oct 31 00:56:35.698687 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.698650 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:35.876367 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.876341 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:56:35.933993 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.933963 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\" (UniqueName: \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\") pod \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " Oct 31 00:56:35.934169 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.934077 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\") pod \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " Oct 31 00:56:35.934169 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.934125 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\") pod \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " Oct 31 00:56:35.934273 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.934196 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") pod \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\" (UID: \"245d7987-2e9d-4aa0-b1c0-1640de7130e9\") " Oct 31 00:56:35.934485 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.934462 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca" pod "245d7987-2e9d-4aa0-b1c0-1640de7130e9" (UID: "245d7987-2e9d-4aa0-b1c0-1640de7130e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:35.934610 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.934587 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245d7987-2e9d-4aa0-b1c0-1640de7130e9-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:35.936188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.936155 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs" pod "245d7987-2e9d-4aa0-b1c0-1640de7130e9" (UID: "245d7987-2e9d-4aa0-b1c0-1640de7130e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:35.936188 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.936172 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf" pod "245d7987-2e9d-4aa0-b1c0-1640de7130e9" (UID: "245d7987-2e9d-4aa0-b1c0-1640de7130e9"). InnerVolumeSpecName "kube-api-access-zdmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:35.936321 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:35.936194 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls" pod "245d7987-2e9d-4aa0-b1c0-1640de7130e9" (UID: "245d7987-2e9d-4aa0-b1c0-1640de7130e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:36.035548 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.035503 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmsf\" (UniqueName: \"kubernetes.io/projected/245d7987-2e9d-4aa0-b1c0-1640de7130e9-kube-api-access-zdmsf\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:36.035726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.035553 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:36.035726 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.035581 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245d7987-2e9d-4aa0-b1c0-1640de7130e9-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:36.342004 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.341968 4471 generic.go:356] "Generic (PLEG): container finished" podID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" containerID="5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35" exitCode=0 Oct 31 00:56:36.342186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.342018 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" Oct 31 00:56:36.342186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.342046 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" event={"ID":"245d7987-2e9d-4aa0-b1c0-1640de7130e9","Type":"ContainerDied","Data":"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35"} Oct 31 00:56:36.342186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.342099 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h" event={"ID":"245d7987-2e9d-4aa0-b1c0-1640de7130e9","Type":"ContainerDied","Data":"f625192d2436be3404d6a5fe8fc377a24018feb12ab538fcd4aadcf8722fca98"} Oct 31 00:56:36.342186 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.342120 4471 scope.go:122] "RemoveContainer" containerID="5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35" Oct 31 00:56:36.354264 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.354247 4471 scope.go:122] "RemoveContainer" containerID="5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35" Oct 31 00:56:36.354650 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:36.354621 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35\": container with ID starting with 5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35 not found: ID does not exist" containerID="5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35" Oct 31 00:56:36.354750 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.354659 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35"} err="failed to get container status \"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35\": rpc error: code = NotFound desc = could not find container \"5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35\": container with ID starting with 5203e42e5074eef917093845950cd0d450e520be8be396166ec167bee1c56c35 not found: ID does not exist" Oct 31 00:56:36.509712 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.509678 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:56:36.567257 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:36.560916 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-101b9e-6fc7b7799c-j848h"] Oct 31 00:56:37.207840 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:37.207807 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" path="/var/lib/kubelet/pods/245d7987-2e9d-4aa0-b1c0-1640de7130e9/volumes" Oct 31 00:56:37.338292 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:37.338259 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.548624 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:37.548551 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:37.704550 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:37.704489 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:38.266568 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:38.266518 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:42.766056 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:42.766020 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:43.111859 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:43.111783 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:43.320709 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:43.320674 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.261090 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:45.261052 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.415651 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:45.415620 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.627894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:45.627704 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:45.629850 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:45.629824 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:56:45.630038 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:45.630019 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" containerID="cri-o://0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7" gracePeriod=30 Oct 31 00:56:47.154347 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:47.154318 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:47.578762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:47.578690 4471 patch_prober.go:29] interesting pod/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:56:47.578762 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:47.578744 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:56:48.764577 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:48.764505 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:49.002109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:49.002077 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:49.175946 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:49.175905 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:49.720126 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:49.720093 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:50.901690 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:50.901627 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:56:51.015097 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015064 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015160 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015198 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015281 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015215 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015332 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015406 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015461 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015433 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015469 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015514 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\") pod \"959c2dea-6fbd-40af-8f9e-063113a88298\" (UID: \"959c2dea-6fbd-40af-8f9e-063113a88298\") " Oct 31 00:56:51.015656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015581 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:51.015656 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015630 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:51.015857 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015796 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:51.016012 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.015991 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.016117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.016015 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.016117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.016031 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.016117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.016043 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:51.016117 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.016040 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:51.017459 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.017438 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:51.017572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.017453 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:51.017572 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.017458 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "kube-api-access-qwtsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:51.017647 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.017569 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy" pod "959c2dea-6fbd-40af-8f9e-063113a88298" (UID: "959c2dea-6fbd-40af-8f9e-063113a88298"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:51.116894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116863 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.116894 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116891 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/959c2dea-6fbd-40af-8f9e-063113a88298-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.117106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116905 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/959c2dea-6fbd-40af-8f9e-063113a88298-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.117106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116918 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtsp\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-kube-api-access-qwtsp\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.117106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116930 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/959c2dea-6fbd-40af-8f9e-063113a88298-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.117106 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.116943 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/959c2dea-6fbd-40af-8f9e-063113a88298-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:51.415500 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.415468 4471 generic.go:356] "Generic (PLEG): container finished" podID="959c2dea-6fbd-40af-8f9e-063113a88298" containerID="0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7" exitCode=0 Oct 31 00:56:51.415727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.415521 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" Oct 31 00:56:51.415727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.415559 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerDied","Data":"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7"} Oct 31 00:56:51.415727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.415631 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl" event={"ID":"959c2dea-6fbd-40af-8f9e-063113a88298","Type":"ContainerDied","Data":"85b052b6a9605ec9a0f8c52c963b99dfd6590047f125e4ac5c64b72e5baa752f"} Oct 31 00:56:51.415727 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.415661 4471 scope.go:122] "RemoveContainer" containerID="0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7" Oct 31 00:56:51.424922 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.424900 4471 scope.go:122] "RemoveContainer" containerID="ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69" Oct 31 00:56:51.432871 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.432843 4471 scope.go:122] "RemoveContainer" containerID="0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7" Oct 31 00:56:51.433135 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:51.433114 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7\": container with ID starting with 0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7 not found: ID does not exist" containerID="0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7" Oct 31 00:56:51.433189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.433146 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7"} err="failed to get container status \"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7\": rpc error: code = NotFound desc = could not find container \"0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7\": container with ID starting with 0a01fbf312e046ac09d0447e8fd658db66091fd12bd967134b9971f2aadb7bf7 not found: ID does not exist" Oct 31 00:56:51.433189 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.433165 4471 scope.go:122] "RemoveContainer" containerID="ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69" Oct 31 00:56:51.433407 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:56:51.433390 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69\": container with ID starting with ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69 not found: ID does not exist" containerID="ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69" Oct 31 00:56:51.433443 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.433414 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69"} err="failed to get container status \"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69\": rpc error: code = NotFound desc = could not find container \"ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69\": container with ID starting with ec71e709fa3dec0d3b2ab4e096b6ea173d23ba9f47faff5ef96e0f01ac76ea69 not found: ID does not exist" Oct 31 00:56:51.585878 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.585796 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:56:51.589717 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:51.589692 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-101b9e-openshift-default-5bbfdd65d5-dz2cl"] Oct 31 00:56:53.209402 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:53.209367 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" path="/var/lib/kubelet/pods/959c2dea-6fbd-40af-8f9e-063113a88298/volumes" Oct 31 00:56:54.260132 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:54.260094 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.757981 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:55.757950 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:55.931519 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:55.931434 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:56.168476 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:56.168438 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:59.276969 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:56:59.276921 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.273759 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:04.273723 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:04.279616 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:57:04.279583 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:57:09.276279 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:09.276229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:14.401285 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:14.401191 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:19.292966 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:19.292928 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:24.190574 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:24.190482 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:29.183864 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:29.183828 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:30.479781 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:30.479744 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:34.087829 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:34.087792 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:34.280703 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:57:34.280664 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:57:39.103847 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:39.103815 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:44.069650 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:44.069599 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:49.072614 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:49.072569 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:54.073270 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:54.073219 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:57:59.213754 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:57:59.213717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:04.283991 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:58:04.283959 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:58:04.284506 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:04.284157 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:09.455175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:09.455097 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:10.652453 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:10.652418 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:14.484242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:14.484202 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:19.580084 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:19.580046 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:24.625682 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:24.625618 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:29.727237 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:29.727196 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:30.915618 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:30.915579 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:34.285395 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:58:34.285362 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:58:34.794556 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:34.794486 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:37.090985 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:37.090947 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:40.023628 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:40.023596 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:45.489552 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:45.489451 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:49.505880 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:49.505838 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:49.698022 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:49.697986 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:50.623626 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:50.623590 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:58:55.635373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:58:55.635335 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:00.683804 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:00.683770 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:04.280870 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:59:04.280834 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:59:05.770729 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:05.770678 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:11.122413 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:11.122374 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:15.612201 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:15.612159 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:15.814373 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:15.814341 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.293386 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:17.293354 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.522844 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:17.522805 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:17.673822 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:17.673786 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:18.368008 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:18.367976 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:22.894049 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:22.893988 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:24.875245 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:24.875210 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:25.064300 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:25.064260 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:25.245441 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:25.245405 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:28.315967 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:28.315932 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:31.383432 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:31.383394 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:33.673400 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:33.673368 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:34.280963 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:59:34.280927 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 00:59:37.564482 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:37.564450 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:59:37.668681 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:37.668643 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 00:59:37.958891 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:37.958841 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:38.139725 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:38.139662 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:38.782280 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:38.782230 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:42.565254 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.565191 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" podUID="fbd92850-978d-4298-8fb0-598f40ae70b0" containerName="payload-processing" containerID="cri-o://ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895" gracePeriod=25 Oct 31 00:59:42.669407 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.669357 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" containerID="cri-o://33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2" gracePeriod=25 Oct 31 00:59:42.815944 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.815882 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:59:42.970897 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.970865 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\") pod \"fbd92850-978d-4298-8fb0-598f40ae70b0\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " Oct 31 00:59:42.971061 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.970981 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\" (UniqueName: \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\") pod \"fbd92850-978d-4298-8fb0-598f40ae70b0\" (UID: \"fbd92850-978d-4298-8fb0-598f40ae70b0\") " Oct 31 00:59:42.971242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.971219 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume" pod "fbd92850-978d-4298-8fb0-598f40ae70b0" (UID: "fbd92850-978d-4298-8fb0-598f40ae70b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:42.971369 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.971353 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd92850-978d-4298-8fb0-598f40ae70b0-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:42.973055 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:42.973025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2" pod "fbd92850-978d-4298-8fb0-598f40ae70b0" (UID: "fbd92850-978d-4298-8fb0-598f40ae70b0"). InnerVolumeSpecName "kube-api-access-jm8l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:43.072438 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.072356 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8l2\" (UniqueName: \"kubernetes.io/projected/fbd92850-978d-4298-8fb0-598f40ae70b0-kube-api-access-jm8l2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:43.178818 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.178780 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:43.250376 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.250341 4471 generic.go:356] "Generic (PLEG): container finished" podID="fbd92850-978d-4298-8fb0-598f40ae70b0" containerID="ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895" exitCode=0 Oct 31 00:59:43.250590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.250386 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" Oct 31 00:59:43.250590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.250423 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" event={"ID":"fbd92850-978d-4298-8fb0-598f40ae70b0","Type":"ContainerDied","Data":"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895"} Oct 31 00:59:43.250590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.250495 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn" event={"ID":"fbd92850-978d-4298-8fb0-598f40ae70b0","Type":"ContainerDied","Data":"6098d07564f3aa20fbe362803012dbbe51bf41a9813791b35aef4ef892f19dca"} Oct 31 00:59:43.250590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.250562 4471 scope.go:122] "RemoveContainer" containerID="ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895" Oct 31 00:59:43.260108 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.260081 4471 scope.go:122] "RemoveContainer" containerID="ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895" Oct 31 00:59:43.260374 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:59:43.260353 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895\": container with ID starting with ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895 not found: ID does not exist" containerID="ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895" Oct 31 00:59:43.260433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.260383 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895"} err="failed to get container status \"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895\": rpc error: code = NotFound desc = could not find container \"ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895\": container with ID starting with ea60df34136d5011b2fef23980eae25910ce5cb58323255628b7fa4027384895 not found: ID does not exist" Oct 31 00:59:43.414927 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.414894 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:59:43.464590 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:43.464553 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc2453-7658dcf9c7-smwvn"] Oct 31 00:59:44.694583 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:44.694494 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:44.985044 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:44.984951 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:45.136960 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:45.136923 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:45.208669 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:45.208636 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd92850-978d-4298-8fb0-598f40ae70b0" path="/var/lib/kubelet/pods/fbd92850-978d-4298-8fb0-598f40ae70b0/volumes" Oct 31 00:59:45.681111 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:45.681037 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:47.689591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:47.689549 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:50.273109 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:50.273078 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:51.814317 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:51.814281 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:52.074372 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:52.074287 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:52.258501 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:52.258458 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:59:52.258745 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:52.258724 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" containerID="cri-o://9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757" gracePeriod=30 Oct 31 00:59:52.307079 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:52.307044 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:53.045718 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:53.045679 4471 patch_prober.go:29] interesting pod/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:59:53.046115 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:53.045747 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:59:53.559787 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:53.559749 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:55.574226 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:55.574189 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 00:59:57.518433 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.518407 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:59:57.623175 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623088 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623348 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623181 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623348 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623232 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623348 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623274 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623348 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623328 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623361 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623426 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623433 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:57.623591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623464 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623591 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623497 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\") pod \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\" (UID: \"9f978cb4-e3aa-4d8e-8580-d3117c16f667\") " Oct 31 00:59:57.623831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623652 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:57.623831 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623699 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:57.623929 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.623913 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:57.624045 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.624024 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:57.624104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.624031 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.624104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.624071 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.624104 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.624088 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.624200 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.624103 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.625344 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.625326 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:57.625452 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.625418 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:59:57.625643 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.625612 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:57.625732 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.625694 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t" pod "9f978cb4-e3aa-4d8e-8580-d3117c16f667" (UID: "9f978cb4-e3aa-4d8e-8580-d3117c16f667"). InnerVolumeSpecName "kube-api-access-7x27t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:57.689638 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.689601 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:59:57.725315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.725284 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.725315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.725310 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.725315 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.725319 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.725589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.725327 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x27t\" (UniqueName: \"kubernetes.io/projected/9f978cb4-e3aa-4d8e-8580-d3117c16f667-kube-api-access-7x27t\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:57.725589 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:57.725336 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f978cb4-e3aa-4d8e-8580-d3117c16f667-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:58.323458 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.323422 4471 generic.go:356] "Generic (PLEG): container finished" podID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerID="9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757" exitCode=0 Oct 31 00:59:58.323649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.323470 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" Oct 31 00:59:58.323649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.323474 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerDied","Data":"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757"} Oct 31 00:59:58.323649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.323518 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n" event={"ID":"9f978cb4-e3aa-4d8e-8580-d3117c16f667","Type":"ContainerDied","Data":"5a9f2805b079227d1a1f31b87c72ab7effa17e928a66d5fd05bdfb1d04628d32"} Oct 31 00:59:58.323649 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.323553 4471 scope.go:122] "RemoveContainer" containerID="9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757" Oct 31 00:59:58.333242 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.333221 4471 scope.go:122] "RemoveContainer" containerID="a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863" Oct 31 00:59:58.341786 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.341771 4471 scope.go:122] "RemoveContainer" containerID="9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757" Oct 31 00:59:58.342021 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:59:58.342000 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757\": container with ID starting with 9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757 not found: ID does not exist" containerID="9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757" Oct 31 00:59:58.342082 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.342028 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757"} err="failed to get container status \"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757\": rpc error: code = NotFound desc = could not find container \"9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757\": container with ID starting with 9ec5b0449dcb595dfe8b2b54f76bd081d5d2d3848c569eb6548798c0cb30a757 not found: ID does not exist" Oct 31 00:59:58.342082 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.342045 4471 scope.go:122] "RemoveContainer" containerID="a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863" Oct 31 00:59:58.342291 ip-10-0-134-11 kubenswrapper[4471]: E1031 00:59:58.342276 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863\": container with ID starting with a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863 not found: ID does not exist" containerID="a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863" Oct 31 00:59:58.342338 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.342295 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863"} err="failed to get container status \"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863\": rpc error: code = NotFound desc = could not find container \"a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863\": container with ID starting with a4f70ebed3ee2754423ca6f6819b4e556279bb7f52f2ec2e5f8a62ba53f4d863 not found: ID does not exist" Oct 31 00:59:58.538349 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.538316 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:59:58.588410 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:58.588330 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bc2453-openshift-default-677bb79dc9-8pn9n"] Oct 31 00:59:59.208206 ip-10-0-134-11 kubenswrapper[4471]: I1031 00:59:59.208171 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" path="/var/lib/kubelet/pods/9f978cb4-e3aa-4d8e-8580-d3117c16f667/volumes" Oct 31 01:00:00.185973 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.185941 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:00.186360 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186329 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:00:00.186360 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186340 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:00:00.186360 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186355 4471 cpu_manager.go:414] "containerMap: removing container" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:00:00.186360 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186360 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186368 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186373 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186381 4471 cpu_manager.go:414] "containerMap: removing container" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186385 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186394 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186399 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186409 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" containerName="maas-api" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186413 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" containerName="maas-api" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186420 4471 cpu_manager.go:414] "containerMap: removing container" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186425 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186436 4471 cpu_manager.go:414] "containerMap: removing container" podUID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" containerName="maas-api" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186440 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" containerName="maas-api" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186445 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186449 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186457 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186461 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186468 4471 cpu_manager.go:414] "containerMap: removing container" podUID="5f11ac15-675a-4de5-be97-fe932e5fa986" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186473 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f11ac15-675a-4de5-be97-fe932e5fa986" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186481 4471 cpu_manager.go:414] "containerMap: removing container" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186485 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerName="main" Oct 31 01:00:00.186489 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186498 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerName="main" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186502 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerName="main" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186509 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186513 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186520 4471 cpu_manager.go:414] "containerMap: removing container" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186524 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186545 4471 cpu_manager.go:414] "containerMap: removing container" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186550 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186557 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186560 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186567 4471 cpu_manager.go:414] "containerMap: removing container" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186571 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186578 4471 cpu_manager.go:414] "containerMap: removing container" podUID="fbd92850-978d-4298-8fb0-598f40ae70b0" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186583 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd92850-978d-4298-8fb0-598f40ae70b0" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186591 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186595 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186604 4471 cpu_manager.go:414] "containerMap: removing container" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186608 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186664 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5849bf-a0ae-46a0-bc80-adbe6bedc03c" containerName="main" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186673 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186679 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fbdefc-9c49-42e5-b842-2ac8bd269ba6" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186685 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa043e5c-378c-43dd-997d-a3c7f0abcd72" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186692 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6908dfb5-e1aa-4e86-97ae-7b29dc14d3b4" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186697 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdac5ef-9fbe-4685-b550-04132790323c" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186702 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="763d31cb-8503-4af7-a2b0-fe264ed5604b" containerName="payload-pre-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186707 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd92850-978d-4298-8fb0-598f40ae70b0" containerName="payload-processing" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186712 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186717 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37f42c5-9658-4fdc-b34f-be9eec95403a" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186722 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186727 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186733 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e3de49-d2d9-4f71-be1a-de0effb425c5" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186738 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186743 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c5e6b5-4557-4aa1-b2b2-4ebdfbd4e6f8" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186749 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8125c08e-1e9b-4ae8-8e73-19e1f44ca3f7" containerName="maas-api" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186755 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="71de909a-5700-4360-8d6d-6551809d6a76" containerName="istio-proxy" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186761 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="245d7987-2e9d-4aa0-b1c0-1640de7130e9" containerName="maas-api" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186766 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efb79f1-cdee-4336-936b-2157801f5e3a" containerName="main" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186771 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f11ac15-675a-4de5-be97-fe932e5fa986" containerName="main" Oct 31 01:00:00.187084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.186777 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b30eb3-3d46-43fa-8fcc-28bfb444ebe3" containerName="main" Oct 31 01:00:00.189854 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.189839 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.250569 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.250540 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:00:00.253002 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.252974 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.253131 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.253067 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjhm\" (UniqueName: \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.302313 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.302282 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:00.354128 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.354099 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.354298 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.354154 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjhm\" (UniqueName: \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.354858 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.354838 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.416452 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.416423 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjhm\" (UniqueName: \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\") pod \"maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.502504 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.502422 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:00.727084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.727051 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:00.882442 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:00.882416 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:00.884640 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:00:00.884614 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e93224_5136_4341_9096_b29d008e0b1d.slice/crio-890ece18dd6225af998d1183f0fc492f538b40d24b96efbe7c33c45b89624b89 WatchSource:0}: Error finding container 890ece18dd6225af998d1183f0fc492f538b40d24b96efbe7c33c45b89624b89: Status 404 returned error can't find the container with id 890ece18dd6225af998d1183f0fc492f538b40d24b96efbe7c33c45b89624b89 Oct 31 01:00:01.343059 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:01.343016 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerStarted","Data":"2b964991404713602e883a69a84be66254e5942c17313755fea5bc2ef8cc670f"} Oct 31 01:00:01.343445 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:01.343106 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerStarted","Data":"890ece18dd6225af998d1183f0fc492f538b40d24b96efbe7c33c45b89624b89"} Oct 31 01:00:01.460363 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:01.460300 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" podStartSLOduration=1.46028282 podStartE2EDuration="1.46028282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:01.458478462 +0000 UTC m=+2728.383747414" watchObservedRunningTime="2026-10-31 01:00:01.46028282 +0000 UTC m=+2728.385551774" Oct 31 01:00:02.349356 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.349320 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3e93224-5136-4341-9096-b29d008e0b1d" containerID="2b964991404713602e883a69a84be66254e5942c17313755fea5bc2ef8cc670f" exitCode=60 Oct 31 01:00:02.349741 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.349403 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerDied","Data":"2b964991404713602e883a69a84be66254e5942c17313755fea5bc2ef8cc670f"} Oct 31 01:00:02.349741 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.349648 4471 scope.go:122] "RemoveContainer" containerID="2b964991404713602e883a69a84be66254e5942c17313755fea5bc2ef8cc670f" Oct 31 01:00:02.916600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.916523 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 01:00:02.981918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.981890 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\") pod \"50929a9c-8263-4806-a654-749734e3aa36\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " Oct 31 01:00:02.982086 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.981930 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\" (UniqueName: \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\") pod \"50929a9c-8263-4806-a654-749734e3aa36\" (UID: \"50929a9c-8263-4806-a654-749734e3aa36\") " Oct 31 01:00:02.982248 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.982227 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume" pod "50929a9c-8263-4806-a654-749734e3aa36" (UID: "50929a9c-8263-4806-a654-749734e3aa36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:02.983975 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:02.983951 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7" pod "50929a9c-8263-4806-a654-749734e3aa36" (UID: "50929a9c-8263-4806-a654-749734e3aa36"). InnerVolumeSpecName "kube-api-access-tzcn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:03.082856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.082823 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50929a9c-8263-4806-a654-749734e3aa36-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:03.082856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.082851 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzcn7\" (UniqueName: \"kubernetes.io/projected/50929a9c-8263-4806-a654-749734e3aa36-kube-api-access-tzcn7\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:03.355433 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.355401 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3e93224-5136-4341-9096-b29d008e0b1d" containerID="08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995" exitCode=60 Oct 31 01:00:03.355875 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.355487 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerDied","Data":"08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995"} Oct 31 01:00:03.355875 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.355579 4471 scope.go:122] "RemoveContainer" containerID="2b964991404713602e883a69a84be66254e5942c17313755fea5bc2ef8cc670f" Oct 31 01:00:03.355990 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.355889 4471 scope.go:122] "RemoveContainer" containerID="08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995" Oct 31 01:00:03.356243 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:03.356210 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft_odh-ai-gateway-infra(d3e93224-5136-4341-9096-b29d008e0b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" Oct 31 01:00:03.356844 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.356813 4471 generic.go:356] "Generic (PLEG): container finished" podID="50929a9c-8263-4806-a654-749734e3aa36" containerID="33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2" exitCode=0 Oct 31 01:00:03.356905 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.356860 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" Oct 31 01:00:03.356951 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.356894 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" event={"ID":"50929a9c-8263-4806-a654-749734e3aa36","Type":"ContainerDied","Data":"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2"} Oct 31 01:00:03.356951 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.356937 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz" event={"ID":"50929a9c-8263-4806-a654-749734e3aa36","Type":"ContainerDied","Data":"6b52de6cfc52387db57fa7c1e2200d743c67fe5c8572d74fa3e94839da164326"} Oct 31 01:00:03.366953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.366938 4471 scope.go:122] "RemoveContainer" containerID="33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2" Oct 31 01:00:03.375201 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.375180 4471 scope.go:122] "RemoveContainer" containerID="33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2" Oct 31 01:00:03.375468 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:03.375447 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2\": container with ID starting with 33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2 not found: ID does not exist" containerID="33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2" Oct 31 01:00:03.375568 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.375474 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2"} err="failed to get container status \"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2\": rpc error: code = NotFound desc = could not find container \"33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2\": container with ID starting with 33f3f48e7bcef82b4974c0e6d8ae04b47619724097843a7f141054c736d9baf2 not found: ID does not exist" Oct 31 01:00:03.599474 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.599395 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 01:00:03.610583 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:03.610547 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc2453-7f45fcd947-x5bwz"] Oct 31 01:00:04.278297 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:04.278264 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:00:04.363211 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:04.363182 4471 scope.go:122] "RemoveContainer" containerID="08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995" Oct 31 01:00:04.363664 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:04.363504 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft_odh-ai-gateway-infra(d3e93224-5136-4341-9096-b29d008e0b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" Oct 31 01:00:04.936264 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:04.936225 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:05.096041 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:05.096003 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:05.208202 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:05.208099 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50929a9c-8263-4806-a654-749734e3aa36" path="/var/lib/kubelet/pods/50929a9c-8263-4806-a654-749734e3aa36/volumes" Oct 31 01:00:05.819085 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:05.819048 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:10.829972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:10.829921 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:15.080092 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:15.080011 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:16.648420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:16.648388 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:16.978522 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:16.978426 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:17.168402 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:17.168350 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:17.664388 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:17.664351 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:18.202308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:18.202276 4471 scope.go:122] "RemoveContainer" containerID="08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995" Oct 31 01:00:19.435152 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:19.435111 4471 generic.go:356] "Generic (PLEG): container finished" podID="d3e93224-5136-4341-9096-b29d008e0b1d" containerID="d5d2b1b086d82d982d17bbd1f5b548a4956bd28edd94b1e3b89d9d487610b162" exitCode=60 Oct 31 01:00:19.435628 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:19.435178 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerDied","Data":"d5d2b1b086d82d982d17bbd1f5b548a4956bd28edd94b1e3b89d9d487610b162"} Oct 31 01:00:19.435628 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:19.435231 4471 scope.go:122] "RemoveContainer" containerID="08053e418e44b4133570c07831039a89f699e3f33b3edabd2c115ead77481995" Oct 31 01:00:19.435628 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:19.435619 4471 scope.go:122] "RemoveContainer" containerID="d5d2b1b086d82d982d17bbd1f5b548a4956bd28edd94b1e3b89d9d487610b162" Oct 31 01:00:19.435911 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:19.435891 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft_odh-ai-gateway-infra(d3e93224-5136-4341-9096-b29d008e0b1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" Oct 31 01:00:20.559933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.559895 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:20.709784 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.709752 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:20.780689 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.780650 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\" (UniqueName: \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\") pod \"d3e93224-5136-4341-9096-b29d008e0b1d\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " Oct 31 01:00:20.780921 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.780719 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\") pod \"d3e93224-5136-4341-9096-b29d008e0b1d\" (UID: \"d3e93224-5136-4341-9096-b29d008e0b1d\") " Oct 31 01:00:20.781109 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.781080 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca" pod "d3e93224-5136-4341-9096-b29d008e0b1d" (UID: "d3e93224-5136-4341-9096-b29d008e0b1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.782809 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.782789 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm" pod "d3e93224-5136-4341-9096-b29d008e0b1d" (UID: "d3e93224-5136-4341-9096-b29d008e0b1d"). InnerVolumeSpecName "kube-api-access-kxjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.882050 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.881968 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjhm\" (UniqueName: \"kubernetes.io/projected/d3e93224-5136-4341-9096-b29d008e0b1d-kube-api-access-kxjhm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.882050 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:20.882000 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3e93224-5136-4341-9096-b29d008e0b1d-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.448479 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:21.448449 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" Oct 31 01:00:21.448479 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:21.448470 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft" event={"ID":"d3e93224-5136-4341-9096-b29d008e0b1d","Type":"ContainerDied","Data":"890ece18dd6225af998d1183f0fc492f538b40d24b96efbe7c33c45b89624b89"} Oct 31 01:00:21.448756 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:21.448554 4471 scope.go:122] "RemoveContainer" containerID="d5d2b1b086d82d982d17bbd1f5b548a4956bd28edd94b1e3b89d9d487610b162" Oct 31 01:00:21.611707 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:21.611620 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:21.616168 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:21.616137 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da9d07-29890140-z9zft"] Oct 31 01:00:22.185378 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:22.185342 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:23.207806 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:23.207774 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" path="/var/lib/kubelet/pods/d3e93224-5136-4341-9096-b29d008e0b1d/volumes" Oct 31 01:00:23.683631 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:23.683587 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:23.829665 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:23.829629 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:24.053156 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:24.053075 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 01:00:24.053341 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:24.053314 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" containerID="cri-o://c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7" gracePeriod=30 Oct 31 01:00:24.062616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:24.062586 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:27.371312 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:27.371278 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:29.300734 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.300706 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 01:00:29.361616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361517 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361576 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361599 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361914 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361707 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361914 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361801 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361914 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361850 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.361914 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361900 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.362102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361937 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.362102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361968 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.362102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.361993 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\") pod \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\" (UID: \"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75\") " Oct 31 01:00:29.362102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362032 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:29.362308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362240 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.362374 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362342 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.362587 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362567 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.362669 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362590 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.362669 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362604 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.362669 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362616 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.362669 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.362622 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.364168 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.364144 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "kube-api-access-njms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.364168 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.364146 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:29.364303 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.364187 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.364435 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.364415 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token" pod "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" (UID: "026a7b63-8e3d-4749-b2e4-a74a5c1c4f75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.463450 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.463418 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njms5\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-kube-api-access-njms5\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.463450 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.463446 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.463450 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.463455 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.463730 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.463464 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.463730 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.463472 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.489819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.489771 4471 generic.go:356] "Generic (PLEG): container finished" podID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerID="c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7" exitCode=0 Oct 31 01:00:29.489999 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.489818 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" Oct 31 01:00:29.489999 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.489833 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerDied","Data":"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7"} Oct 31 01:00:29.489999 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.489890 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp" event={"ID":"026a7b63-8e3d-4749-b2e4-a74a5c1c4f75","Type":"ContainerDied","Data":"75cafcb74067e211d9f30df96b251116da3bb5f11aff715ebf2ae3c232b84fa5"} Oct 31 01:00:29.489999 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.489918 4471 scope.go:122] "RemoveContainer" containerID="c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7" Oct 31 01:00:29.499886 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.499863 4471 scope.go:122] "RemoveContainer" containerID="004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258" Oct 31 01:00:29.508491 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.508470 4471 scope.go:122] "RemoveContainer" containerID="c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7" Oct 31 01:00:29.508848 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:29.508824 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7\": container with ID starting with c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7 not found: ID does not exist" containerID="c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7" Oct 31 01:00:29.508927 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.508857 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7"} err="failed to get container status \"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7\": rpc error: code = NotFound desc = could not find container \"c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7\": container with ID starting with c71e4d97087af19802962f8251288f61583fd838b1f162cdbeb2730a2df10cc7 not found: ID does not exist" Oct 31 01:00:29.508927 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.508876 4471 scope.go:122] "RemoveContainer" containerID="004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258" Oct 31 01:00:29.509149 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:29.509130 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258\": container with ID starting with 004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258 not found: ID does not exist" containerID="004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258" Oct 31 01:00:29.509193 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.509155 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258"} err="failed to get container status \"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258\": rpc error: code = NotFound desc = could not find container \"004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258\": container with ID starting with 004d738b169a585697087f0fc02af657224da609bd248d71b25d7b86d9902258 not found: ID does not exist" Oct 31 01:00:29.654633 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.654599 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 01:00:29.657861 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:29.657838 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4b86e4-openshift-default-79c864d79c-hz7fp"] Oct 31 01:00:31.208452 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:31.208418 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" path="/var/lib/kubelet/pods/026a7b63-8e3d-4749-b2e4-a74a5c1c4f75/volumes" Oct 31 01:00:31.616235 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:31.616146 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:32.595318 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:32.595279 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:34.279998 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:00:34.279957 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:00:34.687602 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:34.687564 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:35.974119 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:35.974066 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:36.806316 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:36.806271 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:36.949948 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:36.949915 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:37.516702 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:37.516662 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:42.506678 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:42.506637 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:46.816501 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:46.816455 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.436903 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:48.436870 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.688434 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:48.688359 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:48.833374 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:48.833338 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:49.397132 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:49.397089 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:54.006146 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:54.006105 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:55.454648 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:55.454615 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:55.658581 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:55.658514 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:55.884902 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:55.884806 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:56.394807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:56.394772 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:00:59.196182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:00:59.196132 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:01.351160 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:01.351119 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 01:01:01.401576 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:01.401504 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 01:01:01.403235 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:01.403205 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 01:01:01.484448 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:01.484413 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:04.279859 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:04.279823 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:01:04.492812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:04.492774 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:08.695130 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:08.695095 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:08.842613 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:08.842566 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:09.488573 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:09.488517 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:16.370925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.370862 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerName="main" containerID="cri-o://7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911" gracePeriod=45 Oct 31 01:01:16.421957 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.421890 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerName="main" containerID="cri-o://d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763" gracePeriod=45 Oct 31 01:01:16.422968 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.422939 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerName="main" containerID="cri-o://3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced" gracePeriod=45 Oct 31 01:01:16.691701 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.691671 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 01:01:16.700102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.700074 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 01:01:16.708838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.708809 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 01:01:16.711393 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.711368 4471 generic.go:356] "Generic (PLEG): container finished" podID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerID="d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763" exitCode=0 Oct 31 01:01:16.711498 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.711404 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" Oct 31 01:01:16.711498 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.711449 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" event={"ID":"d9153ac1-eba6-4adc-8cdd-2c42910be5ad","Type":"ContainerDied","Data":"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763"} Oct 31 01:01:16.711498 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.711486 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm" event={"ID":"d9153ac1-eba6-4adc-8cdd-2c42910be5ad","Type":"ContainerDied","Data":"5846cd8cb1587cf60b708a0f832a69dc70a64ea08530a9f8291ec445024293d1"} Oct 31 01:01:16.711649 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.711508 4471 scope.go:122] "RemoveContainer" containerID="d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763" Oct 31 01:01:16.712574 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.712554 4471 generic.go:356] "Generic (PLEG): container finished" podID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerID="3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced" exitCode=0 Oct 31 01:01:16.712643 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.712583 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" Oct 31 01:01:16.712643 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.712631 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" event={"ID":"8129dfeb-bcb3-4c72-844d-cee5956a8da6","Type":"ContainerDied","Data":"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced"} Oct 31 01:01:16.712735 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.712661 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq" event={"ID":"8129dfeb-bcb3-4c72-844d-cee5956a8da6","Type":"ContainerDied","Data":"c0812c6753fdf24c65f81a52e6698a621fd1ee895c987d1edae94685bd29289d"} Oct 31 01:01:16.713790 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.713769 4471 generic.go:356] "Generic (PLEG): container finished" podID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerID="7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911" exitCode=0 Oct 31 01:01:16.713885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.713815 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" Oct 31 01:01:16.713885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.713848 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" event={"ID":"336d9b30-364f-4901-9337-ac3fdd11b18e","Type":"ContainerDied","Data":"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911"} Oct 31 01:01:16.713885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.713879 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw" event={"ID":"336d9b30-364f-4901-9337-ac3fdd11b18e","Type":"ContainerDied","Data":"3b3512fb92acf185973effe7daf5a6c0ba3e46abdf31d9535d03d0aa5a440828"} Oct 31 01:01:16.719710 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.719693 4471 scope.go:122] "RemoveContainer" containerID="d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763" Oct 31 01:01:16.719984 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:16.719965 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763\": container with ID starting with d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763 not found: ID does not exist" containerID="d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763" Oct 31 01:01:16.720086 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.719996 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763"} err="failed to get container status \"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763\": rpc error: code = NotFound desc = could not find container \"d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763\": container with ID starting with d81a86c44738fb5208930c4fce4c8c6ddb60a44204f9ccfed619f601fee02763 not found: ID does not exist" Oct 31 01:01:16.720086 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.720013 4471 scope.go:122] "RemoveContainer" containerID="3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced" Oct 31 01:01:16.727282 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.727261 4471 scope.go:122] "RemoveContainer" containerID="3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced" Oct 31 01:01:16.727512 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:16.727494 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced\": container with ID starting with 3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced not found: ID does not exist" containerID="3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced" Oct 31 01:01:16.727590 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.727518 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced"} err="failed to get container status \"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced\": rpc error: code = NotFound desc = could not find container \"3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced\": container with ID starting with 3521e3c48a711e9d44ef82be14365c13b60591f8752db8e4d3b0c9fbb8f95ced not found: ID does not exist" Oct 31 01:01:16.727590 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.727551 4471 scope.go:122] "RemoveContainer" containerID="7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911" Oct 31 01:01:16.734982 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.734965 4471 scope.go:122] "RemoveContainer" containerID="7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911" Oct 31 01:01:16.735254 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:16.735235 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911\": container with ID starting with 7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911 not found: ID does not exist" containerID="7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911" Oct 31 01:01:16.735320 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.735262 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911"} err="failed to get container status \"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911\": rpc error: code = NotFound desc = could not find container \"7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911\": container with ID starting with 7d0b4fa24fc68b5e3c6a75130ffde108d312b19f0387d2aa7d70498261912911 not found: ID does not exist" Oct 31 01:01:16.882830 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882749 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\" (UniqueName: \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.882830 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882817 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882846 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882874 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882898 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882928 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.882978 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\" (UniqueName: \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883027 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883080 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\" (UniqueName: \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883127 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883137 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.883139 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883146 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883208 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883248 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\") pod \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\" (UID: \"8129dfeb-bcb3-4c72-844d-cee5956a8da6\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883279 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\" (UniqueName: \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883309 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\" (UniqueName: \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883341 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883343 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883385 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883415 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\") pod \"336d9b30-364f-4901-9337-ac3fdd11b18e\" (UID: \"336d9b30-364f-4901-9337-ac3fdd11b18e\") " Oct 31 01:01:16.883668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883443 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\") pod \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\" (UID: \"d9153ac1-eba6-4adc-8cdd-2c42910be5ad\") " Oct 31 01:01:16.884151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883794 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883830 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.883880 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884053 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884366 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884159 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884445 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884415 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.884747 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884721 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884750 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884765 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884779 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884795 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884809 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884821 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884836 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.884881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.884883 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.885548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.885502 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.885664 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.885631 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.885876 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.885851 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "kube-api-access-skwfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:16.886101 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.886075 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs" pod "8129dfeb-bcb3-4c72-844d-cee5956a8da6" (UID: "8129dfeb-bcb3-4c72-844d-cee5956a8da6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:16.886561 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.886514 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:16.886711 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.886688 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p" pod "d9153ac1-eba6-4adc-8cdd-2c42910be5ad" (UID: "d9153ac1-eba6-4adc-8cdd-2c42910be5ad"). InnerVolumeSpecName "kube-api-access-d692p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:16.886866 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.886846 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:16.887058 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.887040 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "kube-api-access-z95bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:16.887109 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.887092 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs" pod "336d9b30-364f-4901-9337-ac3fdd11b18e" (UID: "336d9b30-364f-4901-9337-ac3fdd11b18e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:16.985592 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985555 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8129dfeb-bcb3-4c72-844d-cee5956a8da6-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985592 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985589 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8129dfeb-bcb3-4c72-844d-cee5956a8da6-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985606 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d692p\" (UniqueName: \"kubernetes.io/projected/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-kube-api-access-d692p\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985621 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/336d9b30-364f-4901-9337-ac3fdd11b18e-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985634 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336d9b30-364f-4901-9337-ac3fdd11b18e-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985647 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95bm\" (UniqueName: \"kubernetes.io/projected/336d9b30-364f-4901-9337-ac3fdd11b18e-kube-api-access-z95bm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985661 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985675 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skwfq\" (UniqueName: \"kubernetes.io/projected/8129dfeb-bcb3-4c72-844d-cee5956a8da6-kube-api-access-skwfq\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:16.985807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:16.985691 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9153ac1-eba6-4adc-8cdd-2c42910be5ad-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:17.209060 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.209031 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 01:01:17.209932 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.209913 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/premium-w4-da9d07-kserve-858f54dcbc-bhrdq"] Oct 31 01:01:17.254697 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.254663 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 01:01:17.254885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.254863 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" containerID="cri-o://315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221" gracePeriod=30 Oct 31 01:01:17.322499 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.322469 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 01:01:17.331186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.331142 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-w4-da9d07-kserve-74c65d4d58-6fvdm"] Oct 31 01:01:17.453987 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.453952 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 01:01:17.462080 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:17.462008 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-da9d07/distinct-2-w4-da9d07-kserve-54457fd465-r7rgw"] Oct 31 01:01:19.207082 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:19.207049 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" path="/var/lib/kubelet/pods/336d9b30-364f-4901-9337-ac3fdd11b18e/volumes" Oct 31 01:01:19.207587 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:19.207424 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" path="/var/lib/kubelet/pods/8129dfeb-bcb3-4c72-844d-cee5956a8da6/volumes" Oct 31 01:01:19.207809 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:19.207795 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" path="/var/lib/kubelet/pods/d9153ac1-eba6-4adc-8cdd-2c42910be5ad/volumes" Oct 31 01:01:22.498369 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.498344 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 01:01:22.529554 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529453 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529554 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529518 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529745 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529580 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529745 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529606 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529745 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529648 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529745 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529682 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529745 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529729 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529988 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529792 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.529988 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529806 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:22.529988 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.529826 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\") pod \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\" (UID: \"9ef373e9-98e7-4e70-a65c-d95a0f84001c\") " Oct 31 01:01:22.530138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530060 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:22.530138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530098 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:22.530138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530104 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:22.530138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530109 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:22.530429 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530332 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.530429 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530354 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.530429 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530369 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.530429 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530383 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.530429 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.530398 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.531805 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.531774 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:22.531938 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.531809 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:22.531995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.531945 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:22.532077 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.532060 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg" pod "9ef373e9-98e7-4e70-a65c-d95a0f84001c" (UID: "9ef373e9-98e7-4e70-a65c-d95a0f84001c"). InnerVolumeSpecName "kube-api-access-76wxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:22.631862 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.631822 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.631862 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.631855 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.631862 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.631866 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.632148 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.631875 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wxg\" (UniqueName: \"kubernetes.io/projected/9ef373e9-98e7-4e70-a65c-d95a0f84001c-kube-api-access-76wxg\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:22.748838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.748805 4471 generic.go:356] "Generic (PLEG): container finished" podID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerID="315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221" exitCode=0 Oct 31 01:01:22.749043 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.748852 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" Oct 31 01:01:22.749043 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.748889 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" event={"ID":"9ef373e9-98e7-4e70-a65c-d95a0f84001c","Type":"ContainerDied","Data":"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221"} Oct 31 01:01:22.749043 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.748942 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds" event={"ID":"9ef373e9-98e7-4e70-a65c-d95a0f84001c","Type":"ContainerDied","Data":"f66b207a8298ea11cb5dc2bfa7fb5e92615f5e80610bc76d7f52e67629f52e90"} Oct 31 01:01:22.749043 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.748965 4471 scope.go:122] "RemoveContainer" containerID="315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221" Oct 31 01:01:22.758790 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.758775 4471 scope.go:122] "RemoveContainer" containerID="315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221" Oct 31 01:01:22.759053 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:22.759036 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221\": container with ID starting with 315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221 not found: ID does not exist" containerID="315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221" Oct 31 01:01:22.759101 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.759062 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221"} err="failed to get container status \"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221\": rpc error: code = NotFound desc = could not find container \"315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221\": container with ID starting with 315251e6cd74c590b9a146071343fd0ab641c9bf95a03c0f99f0fce01060a221 not found: ID does not exist" Oct 31 01:01:22.913543 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.913499 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 01:01:22.918292 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:22.918267 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6m7ds"] Oct 31 01:01:23.207192 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:23.207118 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" path="/var/lib/kubelet/pods/9ef373e9-98e7-4e70-a65c-d95a0f84001c/volumes" Oct 31 01:01:29.188849 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.188810 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 01:01:29.189291 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.189014 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" containerName="maas-api" containerID="cri-o://e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f" gracePeriod=30 Oct 31 01:01:29.301746 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.301678 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:29.507030 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.507002 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 01:01:29.596933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.596894 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\" (UniqueName: \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\") pod \"4345d42b-60f5-4975-945a-68a767f65aa3\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " Oct 31 01:01:29.597149 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.597034 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\") pod \"4345d42b-60f5-4975-945a-68a767f65aa3\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " Oct 31 01:01:29.597149 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.597093 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\") pod \"4345d42b-60f5-4975-945a-68a767f65aa3\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " Oct 31 01:01:29.597255 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.597146 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\") pod \"4345d42b-60f5-4975-945a-68a767f65aa3\" (UID: \"4345d42b-60f5-4975-945a-68a767f65aa3\") " Oct 31 01:01:29.597441 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.597408 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca" pod "4345d42b-60f5-4975-945a-68a767f65aa3" (UID: "4345d42b-60f5-4975-945a-68a767f65aa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:29.599814 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.599767 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l" pod "4345d42b-60f5-4975-945a-68a767f65aa3" (UID: "4345d42b-60f5-4975-945a-68a767f65aa3"). InnerVolumeSpecName "kube-api-access-zbb2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:29.600101 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.600075 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs" pod "4345d42b-60f5-4975-945a-68a767f65aa3" (UID: "4345d42b-60f5-4975-945a-68a767f65aa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:29.601412 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.601382 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls" pod "4345d42b-60f5-4975-945a-68a767f65aa3" (UID: "4345d42b-60f5-4975-945a-68a767f65aa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:29.698589 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.698547 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:29.698589 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.698589 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4345d42b-60f5-4975-945a-68a767f65aa3-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:29.698819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.698606 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbb2l\" (UniqueName: \"kubernetes.io/projected/4345d42b-60f5-4975-945a-68a767f65aa3-kube-api-access-zbb2l\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:29.698819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.698624 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4345d42b-60f5-4975-945a-68a767f65aa3-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:29.786031 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.785929 4471 generic.go:356] "Generic (PLEG): container finished" podID="4345d42b-60f5-4975-945a-68a767f65aa3" containerID="e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f" exitCode=0 Oct 31 01:01:29.786031 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.785979 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" Oct 31 01:01:29.786031 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.785973 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" event={"ID":"4345d42b-60f5-4975-945a-68a767f65aa3","Type":"ContainerDied","Data":"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f"} Oct 31 01:01:29.786341 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.786049 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" event={"ID":"4345d42b-60f5-4975-945a-68a767f65aa3","Type":"ContainerDied","Data":"4c89b8237c9b25abf72e07ab319385c5683a792bc151541a7fdcefc6971f173b"} Oct 31 01:01:29.786341 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.786085 4471 scope.go:122] "RemoveContainer" containerID="e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f" Oct 31 01:01:29.800217 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.799752 4471 scope.go:122] "RemoveContainer" containerID="e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f" Oct 31 01:01:29.800217 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:29.800095 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f\": container with ID starting with e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f not found: ID does not exist" containerID="e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f" Oct 31 01:01:29.800217 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.800129 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f"} err="failed to get container status \"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f\": rpc error: code = NotFound desc = could not find container \"e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f\": container with ID starting with e9adee341d9d7b06922d05eb296f45f36e9fa67b4928433576f81b4bbad0652f not found: ID does not exist" Oct 31 01:01:29.952275 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.952238 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 01:01:29.958803 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:29.958773 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6"] Oct 31 01:01:30.851044 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:30.851007 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:31.207363 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:31.207329 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" path="/var/lib/kubelet/pods/4345d42b-60f5-4975-945a-68a767f65aa3/volumes" Oct 31 01:01:31.222013 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:31.221978 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:31.865339 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:31.865306 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:32.012372 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:32.012337 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:32.381449 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:32.381417 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 01:01:32.381668 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:32.381631 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" containerID="cri-o://7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32" gracePeriod=30 Oct 31 01:01:32.396189 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:32.396146 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da9d07-6966b6d86-jp9r6" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.37:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:01:34.279751 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:34.279710 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:01:37.631060 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.631035 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 01:01:37.680856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.680763 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.680856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.680814 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.680856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.680834 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681097 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.680964 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681097 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681054 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681204 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681096 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681204 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681110 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.681204 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681146 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681204 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681169 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:37.681400 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681208 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681400 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681245 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\") pod \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\" (UID: \"dfa585d6-b179-479c-96d9-bc4c74fbcf33\") " Oct 31 01:01:37.681400 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681377 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.681573 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681419 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.681573 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681382 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.681676 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681643 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.681676 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681665 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.681765 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681681 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.681765 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681695 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.681765 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.681710 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.683114 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.683085 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:37.683114 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.683190 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:37.683482 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.683386 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "kube-api-access-dh2vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:37.683518 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.683506 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token" pod "dfa585d6-b179-479c-96d9-bc4c74fbcf33" (UID: "dfa585d6-b179-479c-96d9-bc4c74fbcf33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:37.782934 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.782897 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.782934 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.782926 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.782934 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.782935 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa585d6-b179-479c-96d9-bc4c74fbcf33-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.783163 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.782947 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2vs\" (UniqueName: \"kubernetes.io/projected/dfa585d6-b179-479c-96d9-bc4c74fbcf33-kube-api-access-dh2vs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:37.823987 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.823955 4471 generic.go:356] "Generic (PLEG): container finished" podID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerID="7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32" exitCode=0 Oct 31 01:01:37.824161 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.824018 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" Oct 31 01:01:37.824161 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.824038 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" event={"ID":"dfa585d6-b179-479c-96d9-bc4c74fbcf33","Type":"ContainerDied","Data":"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32"} Oct 31 01:01:37.824161 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.824092 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q" event={"ID":"dfa585d6-b179-479c-96d9-bc4c74fbcf33","Type":"ContainerDied","Data":"ca40842f17d47e3af107ce5967e7d34f1fd092482beef074f698851e5d8bd8e3"} Oct 31 01:01:37.824161 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.824115 4471 scope.go:122] "RemoveContainer" containerID="7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32" Oct 31 01:01:37.833125 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.833105 4471 scope.go:122] "RemoveContainer" containerID="7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32" Oct 31 01:01:37.833403 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:01:37.833380 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32\": container with ID starting with 7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32 not found: ID does not exist" containerID="7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32" Oct 31 01:01:37.833487 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.833407 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32"} err="failed to get container status \"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32\": rpc error: code = NotFound desc = could not find container \"7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32\": container with ID starting with 7e7e4a9eae442157699d99556a4208849fa54759f94ffef75b94e148bd62aa32 not found: ID does not exist" Oct 31 01:01:37.991831 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:37.991562 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 01:01:38.012218 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:38.012189 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cmr8q"] Oct 31 01:01:39.208198 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.208161 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" path="/var/lib/kubelet/pods/dfa585d6-b179-479c-96d9-bc4c74fbcf33/volumes" Oct 31 01:01:39.355216 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.355180 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:39.508305 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.508217 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:39.672334 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.672286 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:39.716953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.716913 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 01:01:39.717197 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.717169 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" containerID="cri-o://ceb2cf16ad7cd5103967c343c57e0481443500cef85cc3d97bbb4ae32e2354b1" gracePeriod=30 Oct 31 01:01:39.835201 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.835116 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:39.981358 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:39.981325 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:44.860676 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:44.860641 4471 generic.go:356] "Generic (PLEG): container finished" podID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerID="ceb2cf16ad7cd5103967c343c57e0481443500cef85cc3d97bbb4ae32e2354b1" exitCode=0 Oct 31 01:01:44.860676 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:44.860684 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" event={"ID":"e2e1a669-b28b-4878-8e99-59ee07982fce","Type":"ContainerDied","Data":"ceb2cf16ad7cd5103967c343c57e0481443500cef85cc3d97bbb4ae32e2354b1"} Oct 31 01:01:44.966505 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:44.966478 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 01:01:45.051735 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051632 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051735 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051686 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051746 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051778 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051815 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051856 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051905 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.051972 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051954 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.052248 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.051989 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\") pod \"e2e1a669-b28b-4878-8e99-59ee07982fce\" (UID: \"e2e1a669-b28b-4878-8e99-59ee07982fce\") " Oct 31 01:01:45.052248 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052146 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:45.052248 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052157 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:45.052389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052321 4471 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.052389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052339 4471 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-credential-socket\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.052389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052379 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:45.052567 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052475 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:45.052632 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.052568 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:45.053953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.053929 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:01:45.054076 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.054005 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:01:45.054076 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.054009 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:45.054149 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.054099 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc" pod "e2e1a669-b28b-4878-8e99-59ee07982fce" (UID: "e2e1a669-b28b-4878-8e99-59ee07982fce"). InnerVolumeSpecName "kube-api-access-dc7bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:45.153716 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153684 4471 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-workload-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153716 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153712 4471 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2e1a669-b28b-4878-8e99-59ee07982fce-istiod-ca-cert\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153716 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153722 4471 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-data\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153941 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153731 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7bc\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-kube-api-access-dc7bc\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153941 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153741 4471 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-token\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153941 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153749 4471 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-podinfo\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.153941 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.153757 4471 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2e1a669-b28b-4878-8e99-59ee07982fce-istio-envoy\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:45.865910 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.865878 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" Oct 31 01:01:45.866339 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.865922 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw" event={"ID":"e2e1a669-b28b-4878-8e99-59ee07982fce","Type":"ContainerDied","Data":"a5e5c266d11ce0a84e4741f68f0d7d730eff02c3f989f80d8e3ac7b883e8b925"} Oct 31 01:01:45.866339 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.865979 4471 scope.go:122] "RemoveContainer" containerID="ceb2cf16ad7cd5103967c343c57e0481443500cef85cc3d97bbb4ae32e2354b1" Oct 31 01:01:45.884272 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:45.884246 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:01:46.081173 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:46.081133 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 01:01:46.090787 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:46.090756 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da9d07-openshift-default-798d9948bc-s5lkw"] Oct 31 01:01:47.213616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:47.213582 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" path="/var/lib/kubelet/pods/e2e1a669-b28b-4878-8e99-59ee07982fce/volumes" Oct 31 01:01:58.007611 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:01:58.007573 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:04.279605 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:02:04.279567 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:02:17.760588 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.760546 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82"] Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761046 4471 cpu_manager.go:414] "containerMap: removing container" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerName="main" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761067 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerName="main" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761083 4471 cpu_manager.go:414] "containerMap: removing container" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761091 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761112 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761119 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761134 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761141 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761149 4471 cpu_manager.go:414] "containerMap: removing container" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" containerName="maas-api" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761155 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" containerName="maas-api" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761163 4471 cpu_manager.go:414] "containerMap: removing container" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761169 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761178 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761185 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761199 4471 cpu_manager.go:414] "containerMap: removing container" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761206 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761223 4471 cpu_manager.go:414] "containerMap: removing container" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761231 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761243 4471 cpu_manager.go:414] "containerMap: removing container" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761249 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761270 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761277 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761288 4471 cpu_manager.go:414] "containerMap: removing container" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761294 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761304 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761312 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761331 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761338 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47373da-96ae-46fc-be7e-ae1f5f014d7e" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761350 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761357 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761443 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761457 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761467 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="026a7b63-8e3d-4749-b2e4-a74a5c1c4f75" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761476 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="50929a9c-8263-4806-a654-749734e3aa36" containerName="payload-pre-processing" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761485 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="8129dfeb-bcb3-4c72-844d-cee5956a8da6" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761495 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9153ac1-eba6-4adc-8cdd-2c42910be5ad" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761504 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa585d6-b179-479c-96d9-bc4c74fbcf33" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761514 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef373e9-98e7-4e70-a65c-d95a0f84001c" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761523 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f978cb4-e3aa-4d8e-8580-d3117c16f667" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761550 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e1a669-b28b-4878-8e99-59ee07982fce" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761557 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="336d9b30-364f-4901-9337-ac3fdd11b18e" containerName="main" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761567 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="959c2dea-6fbd-40af-8f9e-063113a88298" containerName="istio-proxy" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761578 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:02:17.761918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.761589 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="4345d42b-60f5-4975-945a-68a767f65aa3" containerName="maas-api" Oct 31 01:02:17.764794 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.764775 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.867265 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.867229 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82"] Oct 31 01:02:17.875723 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.875691 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.875891 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.875738 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.875891 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.875839 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.875891 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.875866 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfdd964-00a7-4e31-973d-a35e80995c06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.876083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.875971 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.876083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.876011 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.876083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.876050 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.876083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.876077 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62vvf\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-kube-api-access-62vvf\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.876285 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.876154 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.976817 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.976787 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977020 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.976829 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977020 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.976941 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977020 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.976968 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977020 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.976984 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfdd964-00a7-4e31-973d-a35e80995c06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977020 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977012 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977030 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977056 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977079 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62vvf\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-kube-api-access-62vvf\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977463 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977386 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977519 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977490 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977598 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977574 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977635 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977612 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.977846 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.977822 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebfdd964-00a7-4e31-973d-a35e80995c06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.979245 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.979224 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:17.979608 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:17.979587 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:18.038137 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.038067 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:18.038272 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.038251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62vvf\" (UniqueName: \"kubernetes.io/projected/ebfdd964-00a7-4e31-973d-a35e80995c06-kube-api-access-62vvf\") pod \"maas-default-gateway-openshift-default-6886b96f9-q4c82\" (UID: \"ebfdd964-00a7-4e31-973d-a35e80995c06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:18.077019 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.076982 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:18.495070 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:02:18.495041 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfdd964_00a7_4e31_973d_a35e80995c06.slice/crio-7c66df70333593f54356f2c023e005f39c6f1f3dbf4764c13ff1019d6947c858 WatchSource:0}: Error finding container 7c66df70333593f54356f2c023e005f39c6f1f3dbf4764c13ff1019d6947c858: Status 404 returned error can't find the container with id 7c66df70333593f54356f2c023e005f39c6f1f3dbf4764c13ff1019d6947c858 Oct 31 01:02:18.496305 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.496280 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82"] Oct 31 01:02:18.497751 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.497719 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 01:02:18.497827 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.497784 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 01:02:18.497827 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:18.497811 4471 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 01:02:19.020678 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.020640 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" event={"ID":"ebfdd964-00a7-4e31-973d-a35e80995c06","Type":"ContainerStarted","Data":"e7f0ac8539ab58b6c9d8f1180e882b63969ececfacde8aed46168f8ef9080433"} Oct 31 01:02:19.021143 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.020704 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" event={"ID":"ebfdd964-00a7-4e31-973d-a35e80995c06","Type":"ContainerStarted","Data":"7c66df70333593f54356f2c023e005f39c6f1f3dbf4764c13ff1019d6947c858"} Oct 31 01:02:19.081575 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.081362 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:19.081783 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.081584 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:19.081783 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.081640 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:19.142407 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:19.142348 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podStartSLOduration=2.142333948 podStartE2EDuration="2.142333948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:19.141814815 +0000 UTC m=+2866.067083768" watchObservedRunningTime="2026-10-31 01:02:19.142333948 +0000 UTC m=+2866.067602901" Oct 31 01:02:20.078019 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:20.077978 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:20.078639 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:20.078598 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:20.482937 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:20.482896 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:21.080065 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:21.080021 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:21.080724 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:21.080676 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:22.085699 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:22.085656 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:22.086325 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:22.086293 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:23.078553 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:23.078199 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:23.078553 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:23.078270 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:23.353012 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:23.352929 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:24.078098 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:24.078060 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:24.078393 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:24.078362 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:25.077568 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:25.077510 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-q4c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:02:25.077971 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:25.077581 4471 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" podUID="ebfdd964-00a7-4e31-973d-a35e80995c06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 31 01:02:26.556024 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:26.555981 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:26.556572 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:26.556523 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:26.557365 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:26.557348 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-q4c82" Oct 31 01:02:32.309203 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:32.309166 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:34.280003 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:02:34.279956 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:02:48.431692 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:48.431627 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:48.582019 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:48.581956 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:02:50.415112 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:02:50.415073 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:03.824420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:03:03.824353 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:03.988421 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:03:03.988385 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:04.279663 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:03:04.279622 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:03:32.673453 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:03:32.673416 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:03:34.279833 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:03:34.279798 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:04:04.279803 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:04:04.279765 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:04:13.769059 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:13.769017 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:13.914155 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:13.914120 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.065270 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:23.065231 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.210790 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:23.210747 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.774860 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:23.774806 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:23.924940 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:23.924886 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:33.045083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:33.045030 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:04:34.279719 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:04:34.279677 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:04:46.079970 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.079891 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q"] Oct 31 01:04:46.080453 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.080302 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e93224-5136-4341-9096-b29d008e0b1d" containerName="cleanup" Oct 31 01:04:46.083391 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.083366 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.135662 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.135630 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:46.185597 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.185558 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:04:46.187003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.186977 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.187082 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.187039 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qx4h\" (UniqueName: \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.187128 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.187092 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q"] Oct 31 01:04:46.187177 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.187160 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.287893 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.287848 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qx4h\" (UniqueName: \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.288088 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.287920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.288088 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.287984 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.288616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.288590 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.290267 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.290248 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.349026 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.348943 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qx4h\" (UniqueName: \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\") pod \"maas-api-revoke-sub-d3fba056-8qk7q\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.395885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.395849 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:46.762443 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:04:46.762409 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6dd02c_60ba_44ff_99b9_4d134c030f2c.slice/crio-994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9 WatchSource:0}: Error finding container 994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9: Status 404 returned error can't find the container with id 994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9 Oct 31 01:04:46.763233 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:46.763206 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q"] Oct 31 01:04:47.743008 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:47.742977 4471 generic.go:356] "Generic (PLEG): container finished" podID="0d6dd02c-60ba-44ff-99b9-4d134c030f2c" containerID="65d5aca287fa12966a92e4453cfa831c27fa043744ad24ee2d8c0b1801ff96e5" exitCode=0 Oct 31 01:04:47.743393 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:47.743050 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" event={"ID":"0d6dd02c-60ba-44ff-99b9-4d134c030f2c","Type":"ContainerDied","Data":"65d5aca287fa12966a92e4453cfa831c27fa043744ad24ee2d8c0b1801ff96e5"} Oct 31 01:04:47.743393 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:47.743104 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" event={"ID":"0d6dd02c-60ba-44ff-99b9-4d134c030f2c","Type":"ContainerStarted","Data":"994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9"} Oct 31 01:04:48.900151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:48.900108 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:49.018120 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.018025 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\") pod \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " Oct 31 01:04:49.018287 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.018163 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\" (UniqueName: \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\") pod \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " Oct 31 01:04:49.018287 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.018232 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\") pod \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\" (UID: \"0d6dd02c-60ba-44ff-99b9-4d134c030f2c\") " Oct 31 01:04:49.018688 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.018653 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca" pod "0d6dd02c-60ba-44ff-99b9-4d134c030f2c" (UID: "0d6dd02c-60ba-44ff-99b9-4d134c030f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:49.021886 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.021852 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca" pod "0d6dd02c-60ba-44ff-99b9-4d134c030f2c" (UID: "0d6dd02c-60ba-44ff-99b9-4d134c030f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:49.022447 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.022410 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h" pod "0d6dd02c-60ba-44ff-99b9-4d134c030f2c" (UID: "0d6dd02c-60ba-44ff-99b9-4d134c030f2c"). InnerVolumeSpecName "kube-api-access-6qx4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:49.119993 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.119945 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:49.119993 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.119991 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qx4h\" (UniqueName: \"kubernetes.io/projected/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-kube-api-access-6qx4h\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:49.120300 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.120010 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6dd02c-60ba-44ff-99b9-4d134c030f2c-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:49.754464 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.754434 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" Oct 31 01:04:49.754706 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.754435 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q" event={"ID":"0d6dd02c-60ba-44ff-99b9-4d134c030f2c","Type":"ContainerDied","Data":"994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9"} Oct 31 01:04:49.754706 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:04:49.754578 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="994375c5bdb72e44631f83eaed648da3f60f9e999223e74a3b54578b34f201a9" Oct 31 01:05:04.279794 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:05:04.279756 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:05:33.451598 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:05:33.451556 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:05:34.280088 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:05:34.280052 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:06:04.280273 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:06:04.280235 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:06:33.752294 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:33.752260 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:06:34.279215 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:06:34.279179 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:06:55.841761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.841722 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c"] Oct 31 01:06:55.842267 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.842099 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0d6dd02c-60ba-44ff-99b9-4d134c030f2c" containerName="revoke-keys" Oct 31 01:06:55.842267 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.842110 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6dd02c-60ba-44ff-99b9-4d134c030f2c" containerName="revoke-keys" Oct 31 01:06:55.842267 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.842165 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6dd02c-60ba-44ff-99b9-4d134c030f2c" containerName="revoke-keys" Oct 31 01:06:55.845305 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.845281 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:55.897673 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.897639 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:55.912293 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.912245 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw55\" (UniqueName: \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:55.912454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.912369 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:55.912454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.912428 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:55.947486 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.947453 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:06:55.949022 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:55.949002 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c"] Oct 31 01:06:56.013620 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.013587 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw55\" (UniqueName: \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.013825 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.013636 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.013825 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.013665 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.014343 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.014319 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.015982 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.015960 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.075284 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.075251 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw55\" (UniqueName: \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\") pod \"maas-api-revoke-sub-3d4156f1-29q4c\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.158536 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.158487 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:56.532333 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:06:56.532300 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7fa113_590d_4117_bc0d_a7ac36f9504d.slice/crio-2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539 WatchSource:0}: Error finding container 2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539: Status 404 returned error can't find the container with id 2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539 Oct 31 01:06:56.533559 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:56.533511 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c"] Oct 31 01:06:57.338169 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:57.338131 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" event={"ID":"0f7fa113-590d-4117-bc0d-a7ac36f9504d","Type":"ContainerStarted","Data":"169acf1f6c7568e99a4f3bb9c9274cf9beebf751679775b98c6bf9dd3e81d767"} Oct 31 01:06:57.338594 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:57.338188 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" event={"ID":"0f7fa113-590d-4117-bc0d-a7ac36f9504d","Type":"ContainerStarted","Data":"2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539"} Oct 31 01:06:57.454024 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:57.453976 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" podStartSLOduration=2.453962643 podStartE2EDuration="2.453962643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:57.453659425 +0000 UTC m=+3144.378928378" watchObservedRunningTime="2026-10-31 01:06:57.453962643 +0000 UTC m=+3144.379231596" Oct 31 01:06:58.344269 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:58.344229 4471 generic.go:356] "Generic (PLEG): container finished" podID="0f7fa113-590d-4117-bc0d-a7ac36f9504d" containerID="169acf1f6c7568e99a4f3bb9c9274cf9beebf751679775b98c6bf9dd3e81d767" exitCode=0 Oct 31 01:06:58.344764 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:58.344298 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" event={"ID":"0f7fa113-590d-4117-bc0d-a7ac36f9504d","Type":"ContainerDied","Data":"169acf1f6c7568e99a4f3bb9c9274cf9beebf751679775b98c6bf9dd3e81d767"} Oct 31 01:06:59.474632 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.474609 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:06:59.555442 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.555405 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\") pod \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " Oct 31 01:06:59.555667 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.555457 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\" (UniqueName: \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\") pod \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " Oct 31 01:06:59.555667 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.555493 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\") pod \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\" (UID: \"0f7fa113-590d-4117-bc0d-a7ac36f9504d\") " Oct 31 01:06:59.555901 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.555876 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca" pod "0f7fa113-590d-4117-bc0d-a7ac36f9504d" (UID: "0f7fa113-590d-4117-bc0d-a7ac36f9504d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:59.557545 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.557511 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca" pod "0f7fa113-590d-4117-bc0d-a7ac36f9504d" (UID: "0f7fa113-590d-4117-bc0d-a7ac36f9504d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:59.557654 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.557580 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55" pod "0f7fa113-590d-4117-bc0d-a7ac36f9504d" (UID: "0f7fa113-590d-4117-bc0d-a7ac36f9504d"). InnerVolumeSpecName "kube-api-access-fnw55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:59.657137 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.657109 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:59.657137 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.657135 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnw55\" (UniqueName: \"kubernetes.io/projected/0f7fa113-590d-4117-bc0d-a7ac36f9504d-kube-api-access-fnw55\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:59.657332 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:06:59.657146 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7fa113-590d-4117-bc0d-a7ac36f9504d-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:00.354063 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:00.354034 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" Oct 31 01:07:00.354257 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:00.354033 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c" event={"ID":"0f7fa113-590d-4117-bc0d-a7ac36f9504d","Type":"ContainerDied","Data":"2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539"} Oct 31 01:07:00.354257 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:00.354163 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a298c2b1161de895f68802d73b6fc65d684805c8040fe62bd20d712b837e539" Oct 31 01:07:04.279312 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:07:04.279277 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:07:26.690714 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.690679 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c"] Oct 31 01:07:26.740371 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.740336 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-29q4c"] Oct 31 01:07:26.820516 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.820485 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29"] Oct 31 01:07:26.820882 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.820868 4471 cpu_manager.go:414] "containerMap: removing container" podUID="0f7fa113-590d-4117-bc0d-a7ac36f9504d" containerName="revoke-keys" Oct 31 01:07:26.820925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.820885 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7fa113-590d-4117-bc0d-a7ac36f9504d" containerName="revoke-keys" Oct 31 01:07:26.820970 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.820944 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7fa113-590d-4117-bc0d-a7ac36f9504d" containerName="revoke-keys" Oct 31 01:07:26.823925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.823905 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.854437 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.854404 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.854593 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.854447 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8g7t\" (UniqueName: \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.854593 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.854520 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.876295 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.876273 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:26.876503 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.876487 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:07:26.884353 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.884333 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29"] Oct 31 01:07:26.955123 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.955048 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.955123 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.955086 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8g7t\" (UniqueName: \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.955123 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.955118 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.955732 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.955710 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:26.957370 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:26.957354 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:27.016463 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:27.016430 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8g7t\" (UniqueName: \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\") pod \"maas-api-revoke-sub-3d4156f1-f5w29\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:27.136401 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:27.136366 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:27.208499 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:27.208424 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7fa113-590d-4117-bc0d-a7ac36f9504d" path="/var/lib/kubelet/pods/0f7fa113-590d-4117-bc0d-a7ac36f9504d/volumes" Oct 31 01:07:27.507284 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:07:27.507253 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3a50b8_53ae_4202_b716_d4dbddc46dfe.slice/crio-7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767 WatchSource:0}: Error finding container 7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767: Status 404 returned error can't find the container with id 7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767 Oct 31 01:07:27.508236 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:27.508213 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29"] Oct 31 01:07:28.482586 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:28.482475 4471 generic.go:356] "Generic (PLEG): container finished" podID="2f3a50b8-53ae-4202-b716-d4dbddc46dfe" containerID="40c17f69bc46477f19279fd0e1c553a39f9600558473262e0bfcb2a9c7ef7a50" exitCode=0 Oct 31 01:07:28.482586 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:28.482560 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" event={"ID":"2f3a50b8-53ae-4202-b716-d4dbddc46dfe","Type":"ContainerDied","Data":"40c17f69bc46477f19279fd0e1c553a39f9600558473262e0bfcb2a9c7ef7a50"} Oct 31 01:07:28.483028 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:28.482636 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" event={"ID":"2f3a50b8-53ae-4202-b716-d4dbddc46dfe","Type":"ContainerStarted","Data":"7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767"} Oct 31 01:07:29.610762 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.610738 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:29.679335 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.679305 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\") pod \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " Oct 31 01:07:29.679540 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.679372 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\") pod \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " Oct 31 01:07:29.679540 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.679460 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\" (UniqueName: \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\") pod \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\" (UID: \"2f3a50b8-53ae-4202-b716-d4dbddc46dfe\") " Oct 31 01:07:29.679751 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.679728 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca" pod "2f3a50b8-53ae-4202-b716-d4dbddc46dfe" (UID: "2f3a50b8-53ae-4202-b716-d4dbddc46dfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:29.681438 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.681411 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca" pod "2f3a50b8-53ae-4202-b716-d4dbddc46dfe" (UID: "2f3a50b8-53ae-4202-b716-d4dbddc46dfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:29.681560 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.681472 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t" pod "2f3a50b8-53ae-4202-b716-d4dbddc46dfe" (UID: "2f3a50b8-53ae-4202-b716-d4dbddc46dfe"). InnerVolumeSpecName "kube-api-access-m8g7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:29.780545 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.780435 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8g7t\" (UniqueName: \"kubernetes.io/projected/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-kube-api-access-m8g7t\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:29.780545 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.780468 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:29.780545 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:29.780478 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a50b8-53ae-4202-b716-d4dbddc46dfe-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:30.492883 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:30.492854 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" Oct 31 01:07:30.492883 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:30.492867 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29" event={"ID":"2f3a50b8-53ae-4202-b716-d4dbddc46dfe","Type":"ContainerDied","Data":"7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767"} Oct 31 01:07:30.493084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:30.492912 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d15ff59899556d5f9fff851eae157e0b7cd910e0c73e985247e10c2d4c76767" Oct 31 01:07:34.097939 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:07:34.097897 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:07:34.279550 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:07:34.279477 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:08:04.279722 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:08:04.279639 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:08:25.603795 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.603750 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x"] Oct 31 01:08:25.604290 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.604276 4471 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a50b8-53ae-4202-b716-d4dbddc46dfe" containerName="revoke-keys" Oct 31 01:08:25.604363 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.604296 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a50b8-53ae-4202-b716-d4dbddc46dfe" containerName="revoke-keys" Oct 31 01:08:25.604434 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.604418 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a50b8-53ae-4202-b716-d4dbddc46dfe" containerName="revoke-keys" Oct 31 01:08:25.607388 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.607366 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.659728 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.659697 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:25.692300 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.692262 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.692462 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.692305 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqd6j\" (UniqueName: \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.692462 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.692445 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.709592 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.709558 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:08:25.711082 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.711060 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x"] Oct 31 01:08:25.793557 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.793500 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.793778 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.793575 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqd6j\" (UniqueName: \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.793778 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.793628 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.794217 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.794192 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.795898 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.795875 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.854979 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.854905 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqd6j\" (UniqueName: \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\") pod \"maas-api-revoke-sub-5a29efb3-4p67x\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:25.920003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:25.919971 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:26.283708 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:08:26.283678 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2811d7_40b0_42cf_bb1d_07f25578a0fc.slice/crio-ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b WatchSource:0}: Error finding container ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b: Status 404 returned error can't find the container with id ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b Oct 31 01:08:26.285406 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:26.285380 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x"] Oct 31 01:08:26.743415 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:26.743378 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" event={"ID":"de2811d7-40b0-42cf-bb1d-07f25578a0fc","Type":"ContainerStarted","Data":"2aa66023282a10bc82d75225ec5f08fd2e9b0393940c36c12c47e20c417395d8"} Oct 31 01:08:26.743857 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:26.743433 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" event={"ID":"de2811d7-40b0-42cf-bb1d-07f25578a0fc","Type":"ContainerStarted","Data":"ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b"} Oct 31 01:08:26.860781 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:26.860669 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" podStartSLOduration=1.860648439 podStartE2EDuration="1.860648439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:26.860359904 +0000 UTC m=+3233.785628857" watchObservedRunningTime="2026-10-31 01:08:26.860648439 +0000 UTC m=+3233.785917393" Oct 31 01:08:27.749356 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:27.749323 4471 generic.go:356] "Generic (PLEG): container finished" podID="de2811d7-40b0-42cf-bb1d-07f25578a0fc" containerID="2aa66023282a10bc82d75225ec5f08fd2e9b0393940c36c12c47e20c417395d8" exitCode=0 Oct 31 01:08:27.749749 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:27.749408 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" event={"ID":"de2811d7-40b0-42cf-bb1d-07f25578a0fc","Type":"ContainerDied","Data":"2aa66023282a10bc82d75225ec5f08fd2e9b0393940c36c12c47e20c417395d8"} Oct 31 01:08:28.619002 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:28.618957 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:29.128664 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.128423 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:29.241158 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.241121 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\" (UniqueName: \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\") pod \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " Oct 31 01:08:29.241369 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.241194 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\") pod \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " Oct 31 01:08:29.241369 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.241228 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\") pod \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\" (UID: \"de2811d7-40b0-42cf-bb1d-07f25578a0fc\") " Oct 31 01:08:29.241645 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.241613 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca" pod "de2811d7-40b0-42cf-bb1d-07f25578a0fc" (UID: "de2811d7-40b0-42cf-bb1d-07f25578a0fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:29.244655 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.243889 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j" pod "de2811d7-40b0-42cf-bb1d-07f25578a0fc" (UID: "de2811d7-40b0-42cf-bb1d-07f25578a0fc"). InnerVolumeSpecName "kube-api-access-nqd6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:29.244970 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.244946 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca" pod "de2811d7-40b0-42cf-bb1d-07f25578a0fc" (UID: "de2811d7-40b0-42cf-bb1d-07f25578a0fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:29.342298 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.342189 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqd6j\" (UniqueName: \"kubernetes.io/projected/de2811d7-40b0-42cf-bb1d-07f25578a0fc-kube-api-access-nqd6j\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:29.342298 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.342231 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:29.342298 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.342250 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2811d7-40b0-42cf-bb1d-07f25578a0fc-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:29.786364 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.786327 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" event={"ID":"de2811d7-40b0-42cf-bb1d-07f25578a0fc","Type":"ContainerDied","Data":"ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b"} Oct 31 01:08:29.786710 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.786671 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3eb70bf3a9abadea1867007844d08e9546e99aa45e2fc5f558524c834d362b" Oct 31 01:08:29.786950 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:29.786936 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x" Oct 31 01:08:34.278820 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:08:34.278782 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:08:34.437420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:34.437385 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:08:43.019313 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:43.019269 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29"] Oct 31 01:08:43.069070 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:43.069034 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f5w29"] Oct 31 01:08:43.207881 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:08:43.207846 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3a50b8-53ae-4202-b716-d4dbddc46dfe" path="/var/lib/kubelet/pods/2f3a50b8-53ae-4202-b716-d4dbddc46dfe/volumes" Oct 31 01:09:04.279732 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:09:04.279693 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:09:16.682195 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:16.682114 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:16.835856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:16.835825 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:20.423744 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:20.423705 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:20.594798 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:20.594767 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:20.745321 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:20.745210 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:23.666637 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:23.666600 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:23.831038 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:23.830999 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:24.073182 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.073100 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:09:24.073459 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.073446 4471 cpu_manager.go:414] "containerMap: removing container" podUID="de2811d7-40b0-42cf-bb1d-07f25578a0fc" containerName="revoke-keys" Oct 31 01:09:24.073503 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.073460 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2811d7-40b0-42cf-bb1d-07f25578a0fc" containerName="revoke-keys" Oct 31 01:09:24.073561 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.073554 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2811d7-40b0-42cf-bb1d-07f25578a0fc" containerName="revoke-keys" Oct 31 01:09:24.076499 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.076479 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.156680 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.156647 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9sd\" (UniqueName: \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.156829 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.156688 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.178662 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.178590 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3d7bfc50-dockercfg-2q6l6\"" Oct 31 01:09:24.178662 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.178640 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-863fb0\"" Oct 31 01:09:24.181088 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.180997 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:09:24.258035 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.257975 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9sd\" (UniqueName: \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.258035 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.258045 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.258743 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.258724 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.319365 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.319335 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9sd\" (UniqueName: \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\") pod \"payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.388502 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.388420 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:24.753771 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:09:24.753745 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b32b7bf_a180_47f6_a291_b6fa616820f4.slice/crio-5aad0664a11e4cadbf5142b0089931af0dab45f573648b25631850086cb970ee WatchSource:0}: Error finding container 5aad0664a11e4cadbf5142b0089931af0dab45f573648b25631850086cb970ee: Status 404 returned error can't find the container with id 5aad0664a11e4cadbf5142b0089931af0dab45f573648b25631850086cb970ee Oct 31 01:09:24.756089 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:24.756059 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:09:25.041412 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:25.041324 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" event={"ID":"9b32b7bf-a180-47f6-a291-b6fa616820f4","Type":"ContainerStarted","Data":"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6"} Oct 31 01:09:25.041412 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:25.041373 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" event={"ID":"9b32b7bf-a180-47f6-a291-b6fa616820f4","Type":"ContainerStarted","Data":"5aad0664a11e4cadbf5142b0089931af0dab45f573648b25631850086cb970ee"} Oct 31 01:09:25.041633 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:25.041464 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:25.158389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:25.158337 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" podStartSLOduration=1.158322635 podStartE2EDuration="1.158322635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:25.157011066 +0000 UTC m=+3292.082280020" watchObservedRunningTime="2026-10-31 01:09:25.158322635 +0000 UTC m=+3292.083591588" Oct 31 01:09:27.412812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:27.412783 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.609998 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:27.609968 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:27.753314 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:27.753277 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:32.684491 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:32.684454 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:34.280448 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:09:34.280413 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:09:34.782811 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:34.782774 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:36.048306 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:36.048274 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:09:37.788424 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:37.788377 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:42.905231 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:42.905194 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.212717 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:45.212682 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.823417 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:45.823384 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:45.974660 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:45.974622 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.306162 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.306131 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.632078 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.631989 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:09:48.635737 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.635715 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.732863 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.732827 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:48.737859 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.737813 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-863fb0\"/\"openshift-service-ca.crt\"" Oct 31 01:09:48.737859 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.737818 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-863fb0\"/\"kube-root-ca.crt\"" Oct 31 01:09:48.737859 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.737823 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-863fb0\"/\"default-dockercfg-2gjj5\"" Oct 31 01:09:48.738116 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.737927 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-863fb0\"/\"simulator-single-main-863fb0-kserve-self-signed-certs\"" Oct 31 01:09:48.739758 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.739737 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:09:48.796979 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.796942 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.796979 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.796980 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.797208 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.797083 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.797208 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.797178 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.797208 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.797197 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429c2\" (UniqueName: \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.797318 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.797274 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898306 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898261 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898306 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898305 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898429 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898466 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898491 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429c2\" (UniqueName: \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898569 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898834 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898753 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898834 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898823 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.898943 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.898906 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.900665 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.900644 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.900842 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.900823 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:48.959646 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:48.959610 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429c2\" (UniqueName: \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\") pod \"simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:49.247062 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:49.246979 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:49.615738 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:09:49.615708 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6d0860_decd_4238_9068_c1d81d303b8b.slice/crio-1401a8188db8b55772b66a0057afdc1ec9c6a12474dd424e1c45d1061d8f47fe WatchSource:0}: Error finding container 1401a8188db8b55772b66a0057afdc1ec9c6a12474dd424e1c45d1061d8f47fe: Status 404 returned error can't find the container with id 1401a8188db8b55772b66a0057afdc1ec9c6a12474dd424e1c45d1061d8f47fe Oct 31 01:09:49.616969 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:49.616941 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:09:50.050157 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.050112 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q"] Oct 31 01:09:50.053170 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.053146 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8qk7q"] Oct 31 01:09:50.166359 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.166322 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" event={"ID":"7a6d0860-decd-4238-9068-c1d81d303b8b","Type":"ContainerStarted","Data":"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f"} Oct 31 01:09:50.166515 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.166371 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" event={"ID":"7a6d0860-decd-4238-9068-c1d81d303b8b","Type":"ContainerStarted","Data":"1401a8188db8b55772b66a0057afdc1ec9c6a12474dd424e1c45d1061d8f47fe"} Oct 31 01:09:50.247420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.247391 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:50.283453 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:50.283399 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" podStartSLOduration=2.283385769 podStartE2EDuration="2.283385769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:50.282913728 +0000 UTC m=+3317.208182681" watchObservedRunningTime="2026-10-31 01:09:50.283385769 +0000 UTC m=+3317.208654722" Oct 31 01:09:51.207771 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:51.207731 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6dd02c-60ba-44ff-99b9-4d134c030f2c" path="/var/lib/kubelet/pods/0d6dd02c-60ba-44ff-99b9-4d134c030f2c/volumes" Oct 31 01:09:53.411653 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:53.411617 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:58.498392 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:58.498355 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:09:59.247928 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:59.247893 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:09:59.260314 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:09:59.260289 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:10:00.221436 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:00.221409 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:10:03.675513 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:03.675476 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:04.278854 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:10:04.278820 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:10:08.786066 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:08.786025 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:13.895808 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:13.895769 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:18.994226 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:18.994190 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:24.102483 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:24.102446 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:29.201860 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:29.201822 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:33.730937 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.730900 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv"] Oct 31 01:10:33.734672 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.734653 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.813712 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.813676 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.813888 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.813744 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.813888 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.813810 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64t8\" (UniqueName: \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.836654 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.836625 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:33.836654 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.836650 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:10:33.838257 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.838238 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv"] Oct 31 01:10:33.914544 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.914494 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.914699 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.914581 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64t8\" (UniqueName: \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.914699 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.914637 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.915134 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.915114 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.916950 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.916928 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:33.975710 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:33.975685 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64t8\" (UniqueName: \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\") pod \"maas-api-revoke-sub-de3162a2-nvjsv\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:34.046850 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:34.046783 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:34.279125 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:10:34.279087 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:10:34.347175 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:34.347064 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:34.410505 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:10:34.410477 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e2d560_0940_45bf_9be7_5d139a9c103a.slice/crio-f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4 WatchSource:0}: Error finding container f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4: Status 404 returned error can't find the container with id f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4 Oct 31 01:10:34.411767 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:34.411735 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv"] Oct 31 01:10:35.144885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:35.144851 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:35.375812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:35.375779 4471 generic.go:356] "Generic (PLEG): container finished" podID="e1e2d560-0940-45bf-9be7-5d139a9c103a" containerID="88323bde95fb534a99b9722643fcd8d228b6a4529967bfe2f3dd4d57ec0f560d" exitCode=0 Oct 31 01:10:35.375974 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:35.375852 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" event={"ID":"e1e2d560-0940-45bf-9be7-5d139a9c103a","Type":"ContainerDied","Data":"88323bde95fb534a99b9722643fcd8d228b6a4529967bfe2f3dd4d57ec0f560d"} Oct 31 01:10:35.375974 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:35.375904 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" event={"ID":"e1e2d560-0940-45bf-9be7-5d139a9c103a","Type":"ContainerStarted","Data":"f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4"} Oct 31 01:10:36.505124 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.505099 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:36.535132 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.535106 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\") pod \"e1e2d560-0940-45bf-9be7-5d139a9c103a\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " Oct 31 01:10:36.535263 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.535153 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\") pod \"e1e2d560-0940-45bf-9be7-5d139a9c103a\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " Oct 31 01:10:36.535263 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.535225 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\" (UniqueName: \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\") pod \"e1e2d560-0940-45bf-9be7-5d139a9c103a\" (UID: \"e1e2d560-0940-45bf-9be7-5d139a9c103a\") " Oct 31 01:10:36.535421 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.535401 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca" pod "e1e2d560-0940-45bf-9be7-5d139a9c103a" (UID: "e1e2d560-0940-45bf-9be7-5d139a9c103a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:36.537258 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.537212 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca" pod "e1e2d560-0940-45bf-9be7-5d139a9c103a" (UID: "e1e2d560-0940-45bf-9be7-5d139a9c103a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:36.537411 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.537260 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8" pod "e1e2d560-0940-45bf-9be7-5d139a9c103a" (UID: "e1e2d560-0940-45bf-9be7-5d139a9c103a"). InnerVolumeSpecName "kube-api-access-x64t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:36.636329 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.636265 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64t8\" (UniqueName: \"kubernetes.io/projected/e1e2d560-0940-45bf-9be7-5d139a9c103a-kube-api-access-x64t8\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:36.636329 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.636292 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:36.636329 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:36.636302 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1e2d560-0940-45bf-9be7-5d139a9c103a-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.385913 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:37.385871 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" event={"ID":"e1e2d560-0940-45bf-9be7-5d139a9c103a","Type":"ContainerDied","Data":"f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4"} Oct 31 01:10:37.386084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:37.385931 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8419427c2de4a93b7c6841be74af9147fa7febb9f270ae89948c938bc7d64e4" Oct 31 01:10:37.386084 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:37.385888 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv" Oct 31 01:10:39.392327 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:39.392296 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:44.488468 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:44.488392 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:45.493513 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:45.493473 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:46.007029 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.006994 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w"] Oct 31 01:10:46.007378 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.007365 4471 cpu_manager.go:414] "containerMap: removing container" podUID="e1e2d560-0940-45bf-9be7-5d139a9c103a" containerName="revoke-keys" Oct 31 01:10:46.007421 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.007379 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e2d560-0940-45bf-9be7-5d139a9c103a" containerName="revoke-keys" Oct 31 01:10:46.007460 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.007437 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e2d560-0940-45bf-9be7-5d139a9c103a" containerName="revoke-keys" Oct 31 01:10:46.010794 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.010773 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.063399 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.063372 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:46.073900 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.073874 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:10:46.114503 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.114472 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w"] Oct 31 01:10:46.121212 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.121182 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.121326 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.121239 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.121393 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.121354 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknwx\" (UniqueName: \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.222145 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.222104 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.222350 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.222157 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.222350 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.222185 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknwx\" (UniqueName: \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.222871 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.222852 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.224447 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.224425 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.295927 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.295859 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknwx\" (UniqueName: \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.322761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.322729 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:46.690451 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:10:46.690419 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9368d6d1_b826_46be_b2d1_7ad25723c110.slice/crio-591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161 WatchSource:0}: Error finding container 591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161: Status 404 returned error can't find the container with id 591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161 Oct 31 01:10:46.691453 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:46.691422 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w"] Oct 31 01:10:47.433897 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:47.433856 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" event={"ID":"9368d6d1-b826-46be-b2d1-7ad25723c110","Type":"ContainerStarted","Data":"15c20d668037e44e10a62913390424887994fff486f4eaf0ad7dce2a35e0564a"} Oct 31 01:10:47.434071 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:47.433915 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" event={"ID":"9368d6d1-b826-46be-b2d1-7ad25723c110","Type":"ContainerStarted","Data":"591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161"} Oct 31 01:10:47.553332 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:47.553284 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" podStartSLOduration=2.553269561 podStartE2EDuration="2.553269561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:47.551736154 +0000 UTC m=+3374.477005107" watchObservedRunningTime="2026-10-31 01:10:47.553269561 +0000 UTC m=+3374.478538513" Oct 31 01:10:48.439273 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:48.439241 4471 generic.go:356] "Generic (PLEG): container finished" podID="9368d6d1-b826-46be-b2d1-7ad25723c110" containerID="15c20d668037e44e10a62913390424887994fff486f4eaf0ad7dce2a35e0564a" exitCode=0 Oct 31 01:10:48.439720 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:48.439331 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" event={"ID":"9368d6d1-b826-46be-b2d1-7ad25723c110","Type":"ContainerDied","Data":"15c20d668037e44e10a62913390424887994fff486f4eaf0ad7dce2a35e0564a"} Oct 31 01:10:49.576276 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.576249 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:49.653977 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.653948 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\" (UniqueName: \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\") pod \"9368d6d1-b826-46be-b2d1-7ad25723c110\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " Oct 31 01:10:49.654126 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.654021 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\") pod \"9368d6d1-b826-46be-b2d1-7ad25723c110\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " Oct 31 01:10:49.654126 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.654055 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\") pod \"9368d6d1-b826-46be-b2d1-7ad25723c110\" (UID: \"9368d6d1-b826-46be-b2d1-7ad25723c110\") " Oct 31 01:10:49.654364 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.654331 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca" pod "9368d6d1-b826-46be-b2d1-7ad25723c110" (UID: "9368d6d1-b826-46be-b2d1-7ad25723c110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:49.655992 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.655966 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca" pod "9368d6d1-b826-46be-b2d1-7ad25723c110" (UID: "9368d6d1-b826-46be-b2d1-7ad25723c110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:49.656129 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.656109 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx" pod "9368d6d1-b826-46be-b2d1-7ad25723c110" (UID: "9368d6d1-b826-46be-b2d1-7ad25723c110"). InnerVolumeSpecName "kube-api-access-pknwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:49.754861 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.754788 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pknwx\" (UniqueName: \"kubernetes.io/projected/9368d6d1-b826-46be-b2d1-7ad25723c110-kube-api-access-pknwx\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:49.754861 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.754815 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:49.754861 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:49.754825 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9368d6d1-b826-46be-b2d1-7ad25723c110-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:10:50.091168 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.091024 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:50.451130 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.451104 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" Oct 31 01:10:50.451130 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.451125 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w" event={"ID":"9368d6d1-b826-46be-b2d1-7ad25723c110","Type":"ContainerDied","Data":"591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161"} Oct 31 01:10:50.451347 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.451172 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="591e147c6080356bcec403830ba3748ec9252a62e483f4173cf23a23c79bf161" Oct 31 01:10:50.452079 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.452058 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:10:50.548368 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:50.548318 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:55.288959 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:55.288922 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:57.547254 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:57.547218 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:10:57.802390 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:57.802307 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:57.950443 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:57.950416 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:10:59.801398 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:10:59.801362 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:02.548374 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:02.548320 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" containerID="cri-o://2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6" gracePeriod=25 Oct 31 01:11:04.279094 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:11:04.279053 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:11:05.471775 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.471730 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" podUID="7a6d0860-decd-4238-9068-c1d81d303b8b" containerName="main" containerID="cri-o://8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f" gracePeriod=45 Oct 31 01:11:05.707852 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.707823 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:11:05.816796 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816717 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.816796 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816768 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\" (UniqueName: \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.816995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816825 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.816995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816847 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.816995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816874 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.816995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.816906 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\") pod \"7a6d0860-decd-4238-9068-c1d81d303b8b\" (UID: \"7a6d0860-decd-4238-9068-c1d81d303b8b\") " Oct 31 01:11:05.817200 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817025 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:05.817200 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817141 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:05.817307 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817254 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:05.817413 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817388 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:05.817413 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817414 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:05.817636 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.817428 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:05.819111 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.819079 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:11:05.819111 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.819088 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:05.819239 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.819106 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2" pod "7a6d0860-decd-4238-9068-c1d81d303b8b" (UID: "7a6d0860-decd-4238-9068-c1d81d303b8b"). InnerVolumeSpecName "kube-api-access-429c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:05.918359 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.918325 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429c2\" (UniqueName: \"kubernetes.io/projected/7a6d0860-decd-4238-9068-c1d81d303b8b-kube-api-access-429c2\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:05.918359 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.918355 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6d0860-decd-4238-9068-c1d81d303b8b-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:05.918359 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:05.918365 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6d0860-decd-4238-9068-c1d81d303b8b-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:06.047563 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.047507 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:06.521032 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.520995 4471 generic.go:356] "Generic (PLEG): container finished" podID="7a6d0860-decd-4238-9068-c1d81d303b8b" containerID="8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f" exitCode=0 Oct 31 01:11:06.521512 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.521049 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" Oct 31 01:11:06.521512 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.521077 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" event={"ID":"7a6d0860-decd-4238-9068-c1d81d303b8b","Type":"ContainerDied","Data":"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f"} Oct 31 01:11:06.521512 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.521136 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4" event={"ID":"7a6d0860-decd-4238-9068-c1d81d303b8b","Type":"ContainerDied","Data":"1401a8188db8b55772b66a0057afdc1ec9c6a12474dd424e1c45d1061d8f47fe"} Oct 31 01:11:06.521512 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.521163 4471 scope.go:122] "RemoveContainer" containerID="8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f" Oct 31 01:11:06.529673 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.529655 4471 scope.go:122] "RemoveContainer" containerID="8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f" Oct 31 01:11:06.529954 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:11:06.529934 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f\": container with ID starting with 8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f not found: ID does not exist" containerID="8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f" Oct 31 01:11:06.530029 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.529965 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f"} err="failed to get container status \"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f\": rpc error: code = NotFound desc = could not find container \"8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f\": container with ID starting with 8d711f1d09e5c02911fa12358603b1346ce7ebf1c3196beaa014592d6e86e02f not found: ID does not exist" Oct 31 01:11:06.684849 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.684817 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:11:06.734604 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:06.734574 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-863fb0/simulator-single-main-863fb0-kserve-5bcd8cc887-qk9j4"] Oct 31 01:11:07.207483 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:07.207450 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6d0860-decd-4238-9068-c1d81d303b8b" path="/var/lib/kubelet/pods/7a6d0860-decd-4238-9068-c1d81d303b8b/volumes" Oct 31 01:11:07.628447 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:07.628365 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:07.914851 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:07.914817 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:08.244179 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:08.244099 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w"] Oct 31 01:11:08.293736 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:08.293706 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-863fb0-xxc5w"] Oct 31 01:11:09.002112 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:09.002071 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:09.207567 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:09.207519 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9368d6d1-b826-46be-b2d1-7ad25723c110" path="/var/lib/kubelet/pods/9368d6d1-b826-46be-b2d1-7ad25723c110/volumes" Oct 31 01:11:15.040713 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:15.040674 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:15.186821 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:15.186793 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:15.531847 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:15.531813 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:16.047617 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:16.047571 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:11:17.696682 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.696641 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm"] Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697002 4471 cpu_manager.go:414] "containerMap: removing container" podUID="7a6d0860-decd-4238-9068-c1d81d303b8b" containerName="main" Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697013 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6d0860-decd-4238-9068-c1d81d303b8b" containerName="main" Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697026 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9368d6d1-b826-46be-b2d1-7ad25723c110" containerName="revoke-keys" Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697031 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9368d6d1-b826-46be-b2d1-7ad25723c110" containerName="revoke-keys" Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697099 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6d0860-decd-4238-9068-c1d81d303b8b" containerName="main" Oct 31 01:11:17.697186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.697108 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9368d6d1-b826-46be-b2d1-7ad25723c110" containerName="revoke-keys" Oct 31 01:11:17.701578 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.701554 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.735281 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.735257 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w446j\" (UniqueName: \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.735441 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.735301 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.735441 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.735331 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.804224 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.804164 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:17.804224 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.804203 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:11:17.805871 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.805851 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm"] Oct 31 01:11:17.836859 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.836831 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w446j\" (UniqueName: \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.837003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.836885 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.837003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.836920 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.837501 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.837475 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.839149 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.839130 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:17.897925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:17.897896 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w446j\" (UniqueName: \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\") pod \"maas-api-revoke-sub-3d4156f1-s2zgm\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:18.013845 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:18.013761 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:18.379331 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:11:18.379303 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36f843d_4061_4c8c_a102_27134c8ccdbc.slice/crio-e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931 WatchSource:0}: Error finding container e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931: Status 404 returned error can't find the container with id e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931 Oct 31 01:11:18.380907 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:18.380886 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm"] Oct 31 01:11:18.574985 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:18.574953 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" event={"ID":"d36f843d-4061-4c8c-a102-27134c8ccdbc","Type":"ContainerStarted","Data":"e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931"} Oct 31 01:11:19.580997 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:19.580915 4471 generic.go:356] "Generic (PLEG): container finished" podID="d36f843d-4061-4c8c-a102-27134c8ccdbc" containerID="e078282db7c8a872c35e4e9b5d1ee2b4da044c18a8b4ac2c2b62313438a0c12a" exitCode=0 Oct 31 01:11:19.581431 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:19.580993 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" event={"ID":"d36f843d-4061-4c8c-a102-27134c8ccdbc","Type":"ContainerDied","Data":"e078282db7c8a872c35e4e9b5d1ee2b4da044c18a8b4ac2c2b62313438a0c12a"} Oct 31 01:11:20.709130 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.709107 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:20.764888 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.764858 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\" (UniqueName: \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\") pod \"d36f843d-4061-4c8c-a102-27134c8ccdbc\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " Oct 31 01:11:20.764888 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.764892 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\") pod \"d36f843d-4061-4c8c-a102-27134c8ccdbc\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " Oct 31 01:11:20.765105 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.764915 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\") pod \"d36f843d-4061-4c8c-a102-27134c8ccdbc\" (UID: \"d36f843d-4061-4c8c-a102-27134c8ccdbc\") " Oct 31 01:11:20.765315 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.765287 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca" pod "d36f843d-4061-4c8c-a102-27134c8ccdbc" (UID: "d36f843d-4061-4c8c-a102-27134c8ccdbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:20.766931 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.766902 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca" pod "d36f843d-4061-4c8c-a102-27134c8ccdbc" (UID: "d36f843d-4061-4c8c-a102-27134c8ccdbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:20.767015 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.766938 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j" pod "d36f843d-4061-4c8c-a102-27134c8ccdbc" (UID: "d36f843d-4061-4c8c-a102-27134c8ccdbc"). InnerVolumeSpecName "kube-api-access-w446j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:20.866389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.866317 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w446j\" (UniqueName: \"kubernetes.io/projected/d36f843d-4061-4c8c-a102-27134c8ccdbc-kube-api-access-w446j\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.866389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.866343 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:20.866389 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:20.866355 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36f843d-4061-4c8c-a102-27134c8ccdbc-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:21.591642 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:21.591567 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" Oct 31 01:11:21.591642 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:21.591591 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm" event={"ID":"d36f843d-4061-4c8c-a102-27134c8ccdbc","Type":"ContainerDied","Data":"e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931"} Oct 31 01:11:21.591642 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:21.591643 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95445555866ef85eeca116879de9f74906e010fcfadc8f25a8795e39b275931" Oct 31 01:11:22.791483 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.791460 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:11:22.885810 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.885724 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\" (UniqueName: \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\") pod \"9b32b7bf-a180-47f6-a291-b6fa616820f4\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " Oct 31 01:11:22.885810 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.885787 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\") pod \"9b32b7bf-a180-47f6-a291-b6fa616820f4\" (UID: \"9b32b7bf-a180-47f6-a291-b6fa616820f4\") " Oct 31 01:11:22.886203 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.886173 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume" pod "9b32b7bf-a180-47f6-a291-b6fa616820f4" (UID: "9b32b7bf-a180-47f6-a291-b6fa616820f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:22.887716 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.887695 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd" pod "9b32b7bf-a180-47f6-a291-b6fa616820f4" (UID: "9b32b7bf-a180-47f6-a291-b6fa616820f4"). InnerVolumeSpecName "kube-api-access-mc9sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:22.986845 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.986813 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9sd\" (UniqueName: \"kubernetes.io/projected/9b32b7bf-a180-47f6-a291-b6fa616820f4-kube-api-access-mc9sd\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:22.986845 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:22.986840 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b32b7bf-a180-47f6-a291-b6fa616820f4-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:11:23.602304 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.602217 4471 generic.go:356] "Generic (PLEG): container finished" podID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerID="2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6" exitCode=0 Oct 31 01:11:23.602304 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.602277 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" Oct 31 01:11:23.602496 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.602300 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" event={"ID":"9b32b7bf-a180-47f6-a291-b6fa616820f4","Type":"ContainerDied","Data":"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6"} Oct 31 01:11:23.602496 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.602345 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f" event={"ID":"9b32b7bf-a180-47f6-a291-b6fa616820f4","Type":"ContainerDied","Data":"5aad0664a11e4cadbf5142b0089931af0dab45f573648b25631850086cb970ee"} Oct 31 01:11:23.602496 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.602366 4471 scope.go:122] "RemoveContainer" containerID="2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6" Oct 31 01:11:23.610818 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.610799 4471 scope.go:122] "RemoveContainer" containerID="2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6" Oct 31 01:11:23.611090 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:11:23.611068 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6\": container with ID starting with 2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6 not found: ID does not exist" containerID="2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6" Oct 31 01:11:23.611170 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.611102 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6"} err="failed to get container status \"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6\": rpc error: code = NotFound desc = could not find container \"2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6\": container with ID starting with 2325b4e1d06155fe8f8c709baba664736dc8fbcd23772ebffa25318fa110fdc6 not found: ID does not exist" Oct 31 01:11:23.766205 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.766168 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:11:23.816220 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:23.816178 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-863fb0-6cdb97h84f"] Oct 31 01:11:25.206725 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:25.206680 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" path="/var/lib/kubelet/pods/9b32b7bf-a180-47f6-a291-b6fa616820f4/volumes" Oct 31 01:11:26.605111 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:26.605078 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:26.820360 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:26.820323 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.356722 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:27.356687 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:27.896630 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:27.896592 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:33.768053 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:33.768013 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:11:34.278850 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:11:34.278812 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:11:34.727611 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:34.727579 4471 scope.go:122] "RemoveContainer" containerID="65d5aca287fa12966a92e4453cfa831c27fa043744ad24ee2d8c0b1801ff96e5" Oct 31 01:11:35.508000 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:11:35.507945 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:04.279157 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:12:04.279118 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:12:04.970936 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:12:04.970899 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:34.279778 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:12:34.279739 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:12:35.840921 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:12:35.840885 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:12:36.259590 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:12:36.259554 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:04.281933 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:13:04.281887 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:13:08.444949 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.444913 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h"] Oct 31 01:13:08.445354 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445293 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d36f843d-4061-4c8c-a102-27134c8ccdbc" containerName="revoke-keys" Oct 31 01:13:08.445354 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445305 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36f843d-4061-4c8c-a102-27134c8ccdbc" containerName="revoke-keys" Oct 31 01:13:08.445354 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445320 4471 cpu_manager.go:414] "containerMap: removing container" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" Oct 31 01:13:08.445354 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445325 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" Oct 31 01:13:08.445480 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445389 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36f843d-4061-4c8c-a102-27134c8ccdbc" containerName="revoke-keys" Oct 31 01:13:08.445480 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.445400 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b32b7bf-a180-47f6-a291-b6fa616820f4" containerName="payload-pre-processing" Oct 31 01:13:08.448490 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.448469 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.500935 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.500899 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:08.550802 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.550769 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:13:08.552318 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.552297 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h"] Oct 31 01:13:08.616111 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.616068 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.616320 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.616127 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.616391 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.616343 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7s5x\" (UniqueName: \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.717548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.717439 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7s5x\" (UniqueName: \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.717548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.717483 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.717548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.717549 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.718213 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.718188 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.719833 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.719811 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:08.779253 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:08.779215 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7s5x\" (UniqueName: \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\") pod \"maas-api-revoke-sub-d576e68b-2c75h\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:09.060690 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:09.060601 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:09.438753 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:09.438721 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h"] Oct 31 01:13:09.439471 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:13:09.439443 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf791b43a_de66_4985_ac01_d7c72485c405.slice/crio-7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a WatchSource:0}: Error finding container 7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a: Status 404 returned error can't find the container with id 7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a Oct 31 01:13:10.102023 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:10.101928 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" event={"ID":"f791b43a-de66-4985-ac01-d7c72485c405","Type":"ContainerStarted","Data":"0760197229b35d8389f985f7dbd1c6b3025b294f949fff8e237ed18372e6f635"} Oct 31 01:13:10.102023 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:10.101981 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" event={"ID":"f791b43a-de66-4985-ac01-d7c72485c405","Type":"ContainerStarted","Data":"7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a"} Oct 31 01:13:10.219841 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:10.219782 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" podStartSLOduration=2.2197673780000002 podStartE2EDuration="2.219767378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:10.218804019 +0000 UTC m=+3517.144072973" watchObservedRunningTime="2026-10-31 01:13:10.219767378 +0000 UTC m=+3517.145036330" Oct 31 01:13:11.107761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:11.107725 4471 generic.go:356] "Generic (PLEG): container finished" podID="f791b43a-de66-4985-ac01-d7c72485c405" containerID="0760197229b35d8389f985f7dbd1c6b3025b294f949fff8e237ed18372e6f635" exitCode=0 Oct 31 01:13:11.108217 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:11.107775 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" event={"ID":"f791b43a-de66-4985-ac01-d7c72485c405","Type":"ContainerDied","Data":"0760197229b35d8389f985f7dbd1c6b3025b294f949fff8e237ed18372e6f635"} Oct 31 01:13:12.238500 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.238467 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:12.356353 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.356318 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\" (UniqueName: \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\") pod \"f791b43a-de66-4985-ac01-d7c72485c405\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " Oct 31 01:13:12.356577 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.356379 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\") pod \"f791b43a-de66-4985-ac01-d7c72485c405\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " Oct 31 01:13:12.356577 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.356471 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\") pod \"f791b43a-de66-4985-ac01-d7c72485c405\" (UID: \"f791b43a-de66-4985-ac01-d7c72485c405\") " Oct 31 01:13:12.356862 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.356836 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca" pod "f791b43a-de66-4985-ac01-d7c72485c405" (UID: "f791b43a-de66-4985-ac01-d7c72485c405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:12.358809 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.358781 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca" pod "f791b43a-de66-4985-ac01-d7c72485c405" (UID: "f791b43a-de66-4985-ac01-d7c72485c405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:12.358971 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.358950 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x" pod "f791b43a-de66-4985-ac01-d7c72485c405" (UID: "f791b43a-de66-4985-ac01-d7c72485c405"). InnerVolumeSpecName "kube-api-access-v7s5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:12.457290 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.457239 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:12.457290 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.457286 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7s5x\" (UniqueName: \"kubernetes.io/projected/f791b43a-de66-4985-ac01-d7c72485c405-kube-api-access-v7s5x\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:12.457290 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:12.457297 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f791b43a-de66-4985-ac01-d7c72485c405-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:13.118092 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:13.118063 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" Oct 31 01:13:13.118300 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:13.118097 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h" event={"ID":"f791b43a-de66-4985-ac01-d7c72485c405","Type":"ContainerDied","Data":"7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a"} Oct 31 01:13:13.118300 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:13.118142 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca5e45c0c37577e3e0ef40bd21e548ce1e48ccf71a4d81b34fa2275d6e37c4a" Oct 31 01:13:29.048796 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:29.048759 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x"] Oct 31 01:13:29.105548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:29.105166 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4p67x"] Oct 31 01:13:29.207817 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:29.207776 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2811d7-40b0-42cf-bb1d-07f25578a0fc" path="/var/lib/kubelet/pods/de2811d7-40b0-42cf-bb1d-07f25578a0fc/volumes" Oct 31 01:13:34.279008 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:13:34.278973 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:13:34.761405 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:34.761376 4471 scope.go:122] "RemoveContainer" containerID="40c17f69bc46477f19279fd0e1c553a39f9600558473262e0bfcb2a9c7ef7a50" Oct 31 01:13:34.769729 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:34.769708 4471 scope.go:122] "RemoveContainer" containerID="169acf1f6c7568e99a4f3bb9c9274cf9beebf751679775b98c6bf9dd3e81d767" Oct 31 01:13:36.177762 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:36.177730 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:36.327637 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:36.327605 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.129832 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:50.129755 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:50.838461 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:50.838407 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:51.005871 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:51.005838 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:54.883761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:54.883724 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.058386 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:55.058351 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:55.262141 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:55.262108 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:58.178132 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.178098 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:58.346586 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.346540 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:13:58.584574 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.584473 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:13:58.584889 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.584871 4471 cpu_manager.go:414] "containerMap: removing container" podUID="f791b43a-de66-4985-ac01-d7c72485c405" containerName="revoke-keys" Oct 31 01:13:58.584931 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.584892 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f791b43a-de66-4985-ac01-d7c72485c405" containerName="revoke-keys" Oct 31 01:13:58.584975 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.584966 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="f791b43a-de66-4985-ac01-d7c72485c405" containerName="revoke-keys" Oct 31 01:13:58.587970 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.587951 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.609768 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.609738 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.609904 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.609881 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86vv\" (UniqueName: \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.640260 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.640229 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-abe7bd\"" Oct 31 01:13:58.690233 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.690192 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8ce4f150-dockercfg-4p79p\"" Oct 31 01:13:58.692263 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.692234 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:13:58.711102 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.711070 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86vv\" (UniqueName: \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.711304 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.711108 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.711907 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.711884 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.771977 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.771943 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86vv\" (UniqueName: \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\") pod \"payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:58.900777 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:58.900734 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:59.265494 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:13:59.265452 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd394b94f_1433_4757_a0cc_ed05b0bfebd3.slice/crio-1276a403f12ad39ba63809698bc78ef068e5d76b7d1bece7302135b57454eab3 WatchSource:0}: Error finding container 1276a403f12ad39ba63809698bc78ef068e5d76b7d1bece7302135b57454eab3: Status 404 returned error can't find the container with id 1276a403f12ad39ba63809698bc78ef068e5d76b7d1bece7302135b57454eab3 Oct 31 01:13:59.266443 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:59.266388 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:13:59.343794 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:59.343768 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" event={"ID":"d394b94f-1433-4757-a0cc-ed05b0bfebd3","Type":"ContainerStarted","Data":"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1"} Oct 31 01:13:59.343935 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:59.343821 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" event={"ID":"d394b94f-1433-4757-a0cc-ed05b0bfebd3","Type":"ContainerStarted","Data":"1276a403f12ad39ba63809698bc78ef068e5d76b7d1bece7302135b57454eab3"} Oct 31 01:13:59.343935 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:59.343865 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:13:59.460301 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:13:59.460240 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" podStartSLOduration=1.460223459 podStartE2EDuration="1.460223459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:59.45880613 +0000 UTC m=+3566.384075083" watchObservedRunningTime="2026-10-31 01:13:59.460223459 +0000 UTC m=+3566.385492411" Oct 31 01:14:01.729243 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:01.729198 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:01.876308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:01.876251 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:02.023486 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:02.023387 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:04.279170 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:14:04.279130 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:14:07.077222 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:07.077182 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:10.351546 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:10.351499 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:14:12.295129 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:12.295095 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:17.407300 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:17.407263 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:19.716311 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:19.716270 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.361819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:20.361788 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.518548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:20.518490 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:20.665556 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:20.665511 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:22.908519 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:22.908489 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.320015 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.319935 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:14:23.323323 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.323307 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.371902 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.371870 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-abe7bd\"/\"openshift-service-ca.crt\"" Oct 31 01:14:23.404675 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.404634 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:23.417908 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.417882 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-abe7bd\"/\"simulator-single-main-abe7bd-kserve-self-signed-certs\"" Oct 31 01:14:23.418072 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.417959 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-abe7bd\"/\"kube-root-ca.crt\"" Oct 31 01:14:23.418072 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.417969 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-abe7bd\"/\"default-dockercfg-lvlg2\"" Oct 31 01:14:23.419692 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.419669 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:14:23.444083 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444056 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.444229 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444110 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh46\" (UniqueName: \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.444229 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444219 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.444318 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444283 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.444318 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444311 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.444392 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.444339 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545038 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545001 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545259 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545066 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh46\" (UniqueName: \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545259 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545132 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545259 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545179 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545259 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545212 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545259 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545248 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545501 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545430 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545636 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545615 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.545712 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.545672 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.547391 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.547369 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.547708 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.547690 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.602577 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.602488 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh46\" (UniqueName: \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\") pod \"simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:23.634849 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:23.634817 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:24.005184 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:24.005156 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:14:24.007282 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:14:24.007254 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada4b40e_fe25_488a_af42_ead889f7c426.slice/crio-bf68bdfacce480df11a00f0e09027bc5c6eca43ce0e35af71f87ae92973297f5 WatchSource:0}: Error finding container bf68bdfacce480df11a00f0e09027bc5c6eca43ce0e35af71f87ae92973297f5: Status 404 returned error can't find the container with id bf68bdfacce480df11a00f0e09027bc5c6eca43ce0e35af71f87ae92973297f5 Oct 31 01:14:24.467330 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:24.467296 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" event={"ID":"ada4b40e-fe25-488a-af42-ead889f7c426","Type":"ContainerStarted","Data":"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03"} Oct 31 01:14:24.467330 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:24.467341 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" event={"ID":"ada4b40e-fe25-488a-af42-ead889f7c426","Type":"ContainerStarted","Data":"bf68bdfacce480df11a00f0e09027bc5c6eca43ce0e35af71f87ae92973297f5"} Oct 31 01:14:24.578049 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:24.578001 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" podStartSLOduration=1.577987461 podStartE2EDuration="1.577987461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:24.57673501 +0000 UTC m=+3591.502003962" watchObservedRunningTime="2026-10-31 01:14:24.577987461 +0000 UTC m=+3591.503256412" Oct 31 01:14:24.634949 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:24.634911 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:28.007249 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:28.007174 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.101569 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:33.101511 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:33.635292 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:33.635262 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:33.647399 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:33.647371 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:34.281305 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:14:34.281264 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:14:34.528872 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:34.528842 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:14:34.792961 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:34.792924 4471 scope.go:122] "RemoveContainer" containerID="2aa66023282a10bc82d75225ec5f08fd2e9b0393940c36c12c47e20c417395d8" Oct 31 01:14:36.515804 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:36.515763 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:38.156123 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:38.156091 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:43.279218 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:43.279183 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:48.328733 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:48.328686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:53.400325 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:53.400289 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:14:58.499223 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:14:58.499192 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:00.162953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.162918 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:00.166401 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.166383 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.215202 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.215172 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zfz9z\"" Oct 31 01:15:00.262885 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.262850 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:00.287049 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.287018 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.287213 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.287159 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsxl\" (UniqueName: \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.388015 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.387981 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsxl\" (UniqueName: \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.388209 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.388046 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.388696 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.388668 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.445709 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.445636 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsxl\" (UniqueName: \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\") pod \"maas-api-key-cleanup-29890155-7shqt\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.479378 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.479346 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:00.840460 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:00.840436 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:00.842900 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:15:00.842873 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712ae42f_9f87_42c8_a79d_ce5e87206e2e.slice/crio-9b7e01a85be65e9d1386df51bdbe624a04fd195544cef46ab4d6dc20862fce33 WatchSource:0}: Error finding container 9b7e01a85be65e9d1386df51bdbe624a04fd195544cef46ab4d6dc20862fce33: Status 404 returned error can't find the container with id 9b7e01a85be65e9d1386df51bdbe624a04fd195544cef46ab4d6dc20862fce33 Oct 31 01:15:01.635089 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:01.635004 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerStarted","Data":"15de826f1325b4a8ab3d1e3c06de84973f5a99d0ad2af6fce38d59870a76dde4"} Oct 31 01:15:01.635089 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:01.635061 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerStarted","Data":"9b7e01a85be65e9d1386df51bdbe624a04fd195544cef46ab4d6dc20862fce33"} Oct 31 01:15:01.747208 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:01.747156 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" podStartSLOduration=1.7471436009999999 podStartE2EDuration="1.747143601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:01.746180203 +0000 UTC m=+3628.671449155" watchObservedRunningTime="2026-10-31 01:15:01.747143601 +0000 UTC m=+3628.672412554" Oct 31 01:15:02.641130 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:02.641097 4471 generic.go:356] "Generic (PLEG): container finished" podID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerID="15de826f1325b4a8ab3d1e3c06de84973f5a99d0ad2af6fce38d59870a76dde4" exitCode=60 Oct 31 01:15:02.641645 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:02.641162 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerDied","Data":"15de826f1325b4a8ab3d1e3c06de84973f5a99d0ad2af6fce38d59870a76dde4"} Oct 31 01:15:02.641645 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:02.641454 4471 scope.go:122] "RemoveContainer" containerID="15de826f1325b4a8ab3d1e3c06de84973f5a99d0ad2af6fce38d59870a76dde4" Oct 31 01:15:03.647804 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:03.647767 4471 generic.go:356] "Generic (PLEG): container finished" podID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerID="dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd" exitCode=60 Oct 31 01:15:03.648180 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:03.647844 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerDied","Data":"dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd"} Oct 31 01:15:03.648180 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:03.647898 4471 scope.go:122] "RemoveContainer" containerID="15de826f1325b4a8ab3d1e3c06de84973f5a99d0ad2af6fce38d59870a76dde4" Oct 31 01:15:03.648180 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:03.648142 4471 scope.go:122] "RemoveContainer" containerID="dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd" Oct 31 01:15:03.648400 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:03.648381 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-7shqt_odh-ai-gateway-infra(712ae42f-9f87-42c8-a79d-ce5e87206e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" Oct 31 01:15:03.697516 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:03.697487 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:04.279698 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:04.279663 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:15:04.654319 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:04.654295 4471 scope.go:122] "RemoveContainer" containerID="dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd" Oct 31 01:15:04.654723 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:04.654520 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-7shqt_odh-ai-gateway-infra(712ae42f-9f87-42c8-a79d-ce5e87206e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" Oct 31 01:15:08.779991 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.779955 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:08.867819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.867791 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx"] Oct 31 01:15:08.871138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.871120 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:08.922478 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.922454 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:08.967301 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.967272 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx"] Oct 31 01:15:08.971105 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.971081 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:08.971236 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.971137 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:08.971307 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:08.971287 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzv97\" (UniqueName: \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.072136 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.072051 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.072392 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.072166 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzv97\" (UniqueName: \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.072392 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.072219 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.072848 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.072822 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.074583 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.074563 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.129938 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.129905 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzv97\" (UniqueName: \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\") pod \"maas-api-revoke-sub-41f6ff4c-8sxlx\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.183240 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.183218 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:09.553779 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.553748 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx"] Oct 31 01:15:09.555876 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:15:09.555847 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c4fe7e_3639_412a_931c_5e10cbf9f72f.slice/crio-c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2 WatchSource:0}: Error finding container c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2: Status 404 returned error can't find the container with id c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2 Oct 31 01:15:09.677338 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:09.677296 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" event={"ID":"03c4fe7e-3639-412a-931c-5e10cbf9f72f","Type":"ContainerStarted","Data":"c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2"} Oct 31 01:15:10.683293 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:10.683257 4471 generic.go:356] "Generic (PLEG): container finished" podID="03c4fe7e-3639-412a-931c-5e10cbf9f72f" containerID="2d1aac9c3be060d0dbffbab777fb9af7e6babc91bd47923f3e7272fd4b8f4d31" exitCode=0 Oct 31 01:15:10.683726 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:10.683297 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" event={"ID":"03c4fe7e-3639-412a-931c-5e10cbf9f72f","Type":"ContainerDied","Data":"2d1aac9c3be060d0dbffbab777fb9af7e6babc91bd47923f3e7272fd4b8f4d31"} Oct 31 01:15:11.816706 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.816682 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:11.899405 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.899373 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\") pod \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " Oct 31 01:15:11.899610 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.899497 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\") pod \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " Oct 31 01:15:11.899610 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.899571 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\" (UniqueName: \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\") pod \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\" (UID: \"03c4fe7e-3639-412a-931c-5e10cbf9f72f\") " Oct 31 01:15:11.899861 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.899835 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca" pod "03c4fe7e-3639-412a-931c-5e10cbf9f72f" (UID: "03c4fe7e-3639-412a-931c-5e10cbf9f72f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:11.899932 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.899915 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:11.901556 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.901507 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca" pod "03c4fe7e-3639-412a-931c-5e10cbf9f72f" (UID: "03c4fe7e-3639-412a-931c-5e10cbf9f72f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:11.901687 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:11.901644 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97" pod "03c4fe7e-3639-412a-931c-5e10cbf9f72f" (UID: "03c4fe7e-3639-412a-931c-5e10cbf9f72f"). InnerVolumeSpecName "kube-api-access-nzv97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:12.001319 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:12.001236 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03c4fe7e-3639-412a-931c-5e10cbf9f72f-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.001319 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:12.001266 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzv97\" (UniqueName: \"kubernetes.io/projected/03c4fe7e-3639-412a-931c-5e10cbf9f72f-kube-api-access-nzv97\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:12.693801 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:12.693772 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" Oct 31 01:15:12.693980 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:12.693811 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx" event={"ID":"03c4fe7e-3639-412a-931c-5e10cbf9f72f","Type":"ContainerDied","Data":"c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2"} Oct 31 01:15:12.693980 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:12.693852 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a265c607f6044bea8a801b577d74429d3bfa493715f05816c47d7adfbf66a2" Oct 31 01:15:13.897124 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:13.897042 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:17.202628 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:17.202596 4471 scope.go:122] "RemoveContainer" containerID="dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd" Oct 31 01:15:17.718088 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:17.718009 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerStarted","Data":"aa8e5f4c49d469c68b5938d3881373b244042223d8568aa19ad051411daa11d4"} Oct 31 01:15:18.723938 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.723903 4471 generic.go:356] "Generic (PLEG): container finished" podID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerID="aa8e5f4c49d469c68b5938d3881373b244042223d8568aa19ad051411daa11d4" exitCode=60 Oct 31 01:15:18.724370 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.723950 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerDied","Data":"aa8e5f4c49d469c68b5938d3881373b244042223d8568aa19ad051411daa11d4"} Oct 31 01:15:18.724370 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.724004 4471 scope.go:122] "RemoveContainer" containerID="dc17827f6ad1d5a0ce2923d4bcf38fcbdecede63d1d0a439bc7370fce33d32dd" Oct 31 01:15:18.724370 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.724360 4471 scope.go:122] "RemoveContainer" containerID="aa8e5f4c49d469c68b5938d3881373b244042223d8568aa19ad051411daa11d4" Oct 31 01:15:18.724642 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:18.724622 4471 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-7shqt_odh-ai-gateway-infra(712ae42f-9f87-42c8-a79d-ce5e87206e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" Oct 31 01:15:18.843487 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.843453 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:18.992410 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:18.992321 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:19.879298 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:19.879269 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:19.976430 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:19.976399 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\") pod \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " Oct 31 01:15:19.976621 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:19.976439 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\" (UniqueName: \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\") pod \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\" (UID: \"712ae42f-9f87-42c8-a79d-ce5e87206e2e\") " Oct 31 01:15:19.976751 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:19.976729 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca" pod "712ae42f-9f87-42c8-a79d-ce5e87206e2e" (UID: "712ae42f-9f87-42c8-a79d-ce5e87206e2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.978510 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:19.978493 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl" pod "712ae42f-9f87-42c8-a79d-ce5e87206e2e" (UID: "712ae42f-9f87-42c8-a79d-ce5e87206e2e"). InnerVolumeSpecName "kube-api-access-tjsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.070332 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.070299 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:20.077415 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.077392 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/712ae42f-9f87-42c8-a79d-ce5e87206e2e-maas-api-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.077415 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.077415 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsxl\" (UniqueName: \"kubernetes.io/projected/712ae42f-9f87-42c8-a79d-ce5e87206e2e-kube-api-access-tjsxl\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.576918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.576884 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh"] Oct 31 01:15:20.577253 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577242 4471 cpu_manager.go:414] "containerMap: removing container" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577296 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577254 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577296 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577267 4471 cpu_manager.go:414] "containerMap: removing container" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577296 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577272 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577296 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577283 4471 cpu_manager.go:414] "containerMap: removing container" podUID="03c4fe7e-3639-412a-931c-5e10cbf9f72f" containerName="revoke-keys" Oct 31 01:15:20.577296 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577288 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c4fe7e-3639-412a-931c-5e10cbf9f72f" containerName="revoke-keys" Oct 31 01:15:20.577454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577345 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577353 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c4fe7e-3639-412a-931c-5e10cbf9f72f" containerName="revoke-keys" Oct 31 01:15:20.577454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577364 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.577454 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.577372 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:20.580157 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.580141 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.628847 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.628819 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:20.676726 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.676686 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh"] Oct 31 01:15:20.683280 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.683253 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.683376 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.683331 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlkr\" (UniqueName: \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.683376 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.683354 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.735656 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.735627 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" event={"ID":"712ae42f-9f87-42c8-a79d-ce5e87206e2e","Type":"ContainerDied","Data":"9b7e01a85be65e9d1386df51bdbe624a04fd195544cef46ab4d6dc20862fce33"} Oct 31 01:15:20.735812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.735683 4471 scope.go:122] "RemoveContainer" containerID="aa8e5f4c49d469c68b5938d3881373b244042223d8568aa19ad051411daa11d4" Oct 31 01:15:20.735812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.735635 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt" Oct 31 01:15:20.784399 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.784367 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlkr\" (UniqueName: \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.784525 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.784402 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.784525 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.784448 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.785123 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.785101 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.786594 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.786574 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.841812 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.841754 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlkr\" (UniqueName: \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:20.888747 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.888719 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:20.892386 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.892364 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-7shqt"] Oct 31 01:15:20.897951 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:20.897930 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:21.208015 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:21.207985 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" path="/var/lib/kubelet/pods/712ae42f-9f87-42c8-a79d-ce5e87206e2e/volumes" Oct 31 01:15:21.262210 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:21.262177 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh"] Oct 31 01:15:21.311238 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:15:21.311213 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aa0b11_b2b1_44ac_9c62_30999c71684f.slice/crio-e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747 WatchSource:0}: Error finding container e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747: Status 404 returned error can't find the container with id e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747 Oct 31 01:15:21.743192 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:21.743097 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" event={"ID":"86aa0b11-b2b1-44ac-9c62-30999c71684f","Type":"ContainerStarted","Data":"3e665973423cf4adde81fb7d5df7afbecf3c7661ac436e0a19a9fbefe553fd61"} Oct 31 01:15:21.743192 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:21.743152 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" event={"ID":"86aa0b11-b2b1-44ac-9c62-30999c71684f","Type":"ContainerStarted","Data":"e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747"} Oct 31 01:15:21.851420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:21.851367 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" podStartSLOduration=1.851353333 podStartE2EDuration="1.851353333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:21.850670342 +0000 UTC m=+3648.775939306" watchObservedRunningTime="2026-10-31 01:15:21.851353333 +0000 UTC m=+3648.776622285" Oct 31 01:15:22.752133 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:22.752094 4471 generic.go:356] "Generic (PLEG): container finished" podID="86aa0b11-b2b1-44ac-9c62-30999c71684f" containerID="3e665973423cf4adde81fb7d5df7afbecf3c7661ac436e0a19a9fbefe553fd61" exitCode=0 Oct 31 01:15:22.752522 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:22.752162 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" event={"ID":"86aa0b11-b2b1-44ac-9c62-30999c71684f","Type":"ContainerDied","Data":"3e665973423cf4adde81fb7d5df7afbecf3c7661ac436e0a19a9fbefe553fd61"} Oct 31 01:15:23.887879 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:23.887852 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:24.011762 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.011678 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\") pod \"86aa0b11-b2b1-44ac-9c62-30999c71684f\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " Oct 31 01:15:24.011925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.011814 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\") pod \"86aa0b11-b2b1-44ac-9c62-30999c71684f\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " Oct 31 01:15:24.011925 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.011879 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\" (UniqueName: \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\") pod \"86aa0b11-b2b1-44ac-9c62-30999c71684f\" (UID: \"86aa0b11-b2b1-44ac-9c62-30999c71684f\") " Oct 31 01:15:24.012169 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.012145 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca" pod "86aa0b11-b2b1-44ac-9c62-30999c71684f" (UID: "86aa0b11-b2b1-44ac-9c62-30999c71684f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:24.013838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.013818 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr" pod "86aa0b11-b2b1-44ac-9c62-30999c71684f" (UID: "86aa0b11-b2b1-44ac-9c62-30999c71684f"). InnerVolumeSpecName "kube-api-access-xxlkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:24.013907 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.013876 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca" pod "86aa0b11-b2b1-44ac-9c62-30999c71684f" (UID: "86aa0b11-b2b1-44ac-9c62-30999c71684f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:24.113385 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.113354 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-openshift-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:24.113385 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.113380 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlkr\" (UniqueName: \"kubernetes.io/projected/86aa0b11-b2b1-44ac-9c62-30999c71684f-kube-api-access-xxlkr\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:24.113588 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.113391 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86aa0b11-b2b1-44ac-9c62-30999c71684f-maas-api-xks-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:24.586852 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.586816 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:24.762737 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.762652 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" Oct 31 01:15:24.762737 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.762700 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh" event={"ID":"86aa0b11-b2b1-44ac-9c62-30999c71684f","Type":"ContainerDied","Data":"e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747"} Oct 31 01:15:24.762969 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:24.762758 4471 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e769e32036166a863c0f7437f25737db1e91092b3617a1c5afb8656fd1a1a747" Oct 31 01:15:25.024305 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:25.024230 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:15:25.125337 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:25.125306 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:29.783327 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:29.783289 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:32.123219 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:32.123186 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:15:32.375202 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:32.375087 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:32.522432 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:32.522371 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:34.254220 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:34.254181 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:34.281815 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:34.281775 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:15:36.923967 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:36.923935 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:37.046482 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:37.046448 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv"] Oct 31 01:15:37.092485 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:37.092453 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de3162a2-nvjsv"] Oct 31 01:15:37.123938 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:37.123893 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" containerID="cri-o://6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1" gracePeriod=25 Oct 31 01:15:37.207780 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:37.207704 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e2d560-0940-45bf-9be7-5d139a9c103a" path="/var/lib/kubelet/pods/e1e2d560-0940-45bf-9be7-5d139a9c103a/volumes" Oct 31 01:15:37.264313 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:37.264271 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:40.044122 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.044086 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" podUID="ada4b40e-fe25-488a-af42-ead889f7c426" containerName="main" containerID="cri-o://45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03" gracePeriod=45 Oct 31 01:15:40.287135 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.287106 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:15:40.350253 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.350174 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:40.358502 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358479 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\" (UniqueName: \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358592 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358544 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358725 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358708 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358768 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358751 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\" (UniqueName: \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358811 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358781 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358846 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358812 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\") pod \"ada4b40e-fe25-488a-af42-ead889f7c426\" (UID: \"ada4b40e-fe25-488a-af42-ead889f7c426\") " Oct 31 01:15:40.358846 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358780 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:40.359001 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358971 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:40.359061 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.358982 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:40.359107 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.359097 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.359151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.359113 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.359151 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.359121 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.360801 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.360765 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:40.360907 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.360867 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:40.360907 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.360888 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46" pod "ada4b40e-fe25-488a-af42-ead889f7c426" (UID: "ada4b40e-fe25-488a-af42-ead889f7c426"). InnerVolumeSpecName "kube-api-access-6kh46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:40.460063 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.460029 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh46\" (UniqueName: \"kubernetes.io/projected/ada4b40e-fe25-488a-af42-ead889f7c426-kube-api-access-6kh46\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.460063 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.460059 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada4b40e-fe25-488a-af42-ead889f7c426-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.460063 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.460069 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada4b40e-fe25-488a-af42-ead889f7c426-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:40.837391 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.837355 4471 generic.go:356] "Generic (PLEG): container finished" podID="ada4b40e-fe25-488a-af42-ead889f7c426" containerID="45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03" exitCode=0 Oct 31 01:15:40.837600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.837399 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" Oct 31 01:15:40.837600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.837438 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" event={"ID":"ada4b40e-fe25-488a-af42-ead889f7c426","Type":"ContainerDied","Data":"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03"} Oct 31 01:15:40.837600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.837480 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5" event={"ID":"ada4b40e-fe25-488a-af42-ead889f7c426","Type":"ContainerDied","Data":"bf68bdfacce480df11a00f0e09027bc5c6eca43ce0e35af71f87ae92973297f5"} Oct 31 01:15:40.837600 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.837497 4471 scope.go:122] "RemoveContainer" containerID="45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03" Oct 31 01:15:40.846321 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.846304 4471 scope.go:122] "RemoveContainer" containerID="45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03" Oct 31 01:15:40.846670 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:40.846648 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03\": container with ID starting with 45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03 not found: ID does not exist" containerID="45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03" Oct 31 01:15:40.846761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.846682 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03"} err="failed to get container status \"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03\": rpc error: code = NotFound desc = could not find container \"45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03\": container with ID starting with 45587f68ad9d80bf3c786f38f55863779e82ca8deb63563373cced964055df03 not found: ID does not exist" Oct 31 01:15:40.989817 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:40.989781 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:15:41.036043 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:41.036015 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-abe7bd/simulator-single-main-abe7bd-kserve-7778ff5455-9mqj5"] Oct 31 01:15:41.207725 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:41.207689 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada4b40e-fe25-488a-af42-ead889f7c426" path="/var/lib/kubelet/pods/ada4b40e-fe25-488a-af42-ead889f7c426/volumes" Oct 31 01:15:42.039316 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:42.039284 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:42.322086 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:42.322006 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:42.672539 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:42.672508 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh"] Oct 31 01:15:42.718303 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:42.718275 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-abe7bd-zf4bh"] Oct 31 01:15:42.903277 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:42.903245 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:43.173732 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:43.173697 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:43.207273 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:43.207243 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86aa0b11-b2b1-44ac-9c62-30999c71684f" path="/var/lib/kubelet/pods/86aa0b11-b2b1-44ac-9c62-30999c71684f/volumes" Oct 31 01:15:49.424260 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:49.424227 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.573502 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:49.573471 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:49.763190 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:49.763092 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:50.350541 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:50.350489 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:15:56.499382 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499347 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499727 4471 cpu_manager.go:414] "containerMap: removing container" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499739 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712ae42f-9f87-42c8-a79d-ce5e87206e2e" containerName="cleanup" Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499768 4471 cpu_manager.go:414] "containerMap: removing container" podUID="ada4b40e-fe25-488a-af42-ead889f7c426" containerName="main" Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499774 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada4b40e-fe25-488a-af42-ead889f7c426" containerName="main" Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499782 4471 cpu_manager.go:414] "containerMap: removing container" podUID="86aa0b11-b2b1-44ac-9c62-30999c71684f" containerName="revoke-keys" Oct 31 01:15:56.499789 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499788 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86aa0b11-b2b1-44ac-9c62-30999c71684f" containerName="revoke-keys" Oct 31 01:15:56.499979 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499849 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada4b40e-fe25-488a-af42-ead889f7c426" containerName="main" Oct 31 01:15:56.499979 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.499857 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="86aa0b11-b2b1-44ac-9c62-30999c71684f" containerName="revoke-keys" Oct 31 01:15:56.504400 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.504377 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.553127 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.553095 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 01:15:56.584992 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.584954 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:15:56.598936 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.598912 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 01:15:56.599077 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.598990 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hg9vc\"" Oct 31 01:15:56.599077 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.599025 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-df3d3673-kserve-self-signed-certs\"" Oct 31 01:15:56.600929 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.600908 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:15:56.612684 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612656 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.612808 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612784 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwrr\" (UniqueName: \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.612852 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612843 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.612895 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612864 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.612944 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612928 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.613011 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.612994 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714578 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714509 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714782 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714625 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwrr\" (UniqueName: \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714782 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714686 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714782 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714710 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714782 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714732 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.714782 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.714757 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.715166 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.715138 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.715270 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.715197 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.715270 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.715234 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.717301 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.717279 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.717467 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.717451 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.772258 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.772182 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwrr\" (UniqueName: \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\") pod \"e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:56.815872 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:56.815827 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:57.190982 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.190898 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:15:57.193583 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:15:57.193517 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45fe0be3_7c04_4752_8587_23a463343a55.slice/crio-ac7eeb77640bdbd59529dfd7672f0caa8791246d7c09c2363f6910a3713f4c79 WatchSource:0}: Error finding container ac7eeb77640bdbd59529dfd7672f0caa8791246d7c09c2363f6910a3713f4c79: Status 404 returned error can't find the container with id ac7eeb77640bdbd59529dfd7672f0caa8791246d7c09c2363f6910a3713f4c79 Oct 31 01:15:57.357787 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.357759 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:15:57.522867 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.522834 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\") pod \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " Oct 31 01:15:57.523271 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.522911 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\" (UniqueName: \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\") pod \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\" (UID: \"d394b94f-1433-4757-a0cc-ed05b0bfebd3\") " Oct 31 01:15:57.523271 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.523226 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume" pod "d394b94f-1433-4757-a0cc-ed05b0bfebd3" (UID: "d394b94f-1433-4757-a0cc-ed05b0bfebd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:57.525027 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.525004 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv" pod "d394b94f-1433-4757-a0cc-ed05b0bfebd3" (UID: "d394b94f-1433-4757-a0cc-ed05b0bfebd3"). InnerVolumeSpecName "kube-api-access-w86vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:57.624493 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.624462 4471 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d394b94f-1433-4757-a0cc-ed05b0bfebd3-plugins-config-volume\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:57.624493 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.624489 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86vv\" (UniqueName: \"kubernetes.io/projected/d394b94f-1433-4757-a0cc-ed05b0bfebd3-kube-api-access-w86vv\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:57.918704 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.918673 4471 generic.go:356] "Generic (PLEG): container finished" podID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerID="6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1" exitCode=0 Oct 31 01:15:57.918880 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.918725 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" Oct 31 01:15:57.918880 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.918757 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" event={"ID":"d394b94f-1433-4757-a0cc-ed05b0bfebd3","Type":"ContainerDied","Data":"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1"} Oct 31 01:15:57.918880 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.918820 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw" event={"ID":"d394b94f-1433-4757-a0cc-ed05b0bfebd3","Type":"ContainerDied","Data":"1276a403f12ad39ba63809698bc78ef068e5d76b7d1bece7302135b57454eab3"} Oct 31 01:15:57.918880 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.918845 4471 scope.go:122] "RemoveContainer" containerID="6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1" Oct 31 01:15:57.920373 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.920354 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" event={"ID":"45fe0be3-7c04-4752-8587-23a463343a55","Type":"ContainerStarted","Data":"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958"} Oct 31 01:15:57.920452 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.920383 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" event={"ID":"45fe0be3-7c04-4752-8587-23a463343a55","Type":"ContainerStarted","Data":"ac7eeb77640bdbd59529dfd7672f0caa8791246d7c09c2363f6910a3713f4c79"} Oct 31 01:15:57.927845 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.927829 4471 scope.go:122] "RemoveContainer" containerID="6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1" Oct 31 01:15:57.928120 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:15:57.928095 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1\": container with ID starting with 6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1 not found: ID does not exist" containerID="6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1" Oct 31 01:15:57.928221 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:57.928126 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1"} err="failed to get container status \"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1\": rpc error: code = NotFound desc = could not find container \"6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1\": container with ID starting with 6341d5f0681f41dcbc10abd9dae52eb854a080d6cb14ffcc1c8106a8c5eab7e1 not found: ID does not exist" Oct 31 01:15:58.071573 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:58.071540 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:15:58.117426 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:58.117399 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-abe7bd-545c4j4fxw"] Oct 31 01:15:58.182773 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:58.182672 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" podStartSLOduration=2.182658513 podStartE2EDuration="2.182658513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:58.182081501 +0000 UTC m=+3685.107350454" watchObservedRunningTime="2026-10-31 01:15:58.182658513 +0000 UTC m=+3685.107927466" Oct 31 01:15:58.816265 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:58.816226 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:15:59.206667 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:15:59.206635 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" path="/var/lib/kubelet/pods/d394b94f-1433-4757-a0cc-ed05b0bfebd3/volumes" Oct 31 01:16:02.462933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:02.462892 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:02.806602 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:02.805403 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:04.295719 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:04.295680 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:16:04.296330 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:04.296276 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:16:04.328123 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:16:04.328050 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:16:06.816470 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:06.816427 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:16:06.828960 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:06.828935 4471 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:16:06.973378 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:06.973351 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:16:17.651456 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:17.651423 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:21.046332 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:21.046297 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm"] Oct 31 01:16:21.092180 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:21.092146 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2zgm"] Oct 31 01:16:21.206793 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:21.206756 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36f843d-4061-4c8c-a102-27134c8ccdbc" path="/var/lib/kubelet/pods/d36f843d-4061-4c8c-a102-27134c8ccdbc/volumes" Oct 31 01:16:34.279261 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:16:34.279226 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:16:34.825273 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:34.825243 4471 scope.go:122] "RemoveContainer" containerID="88323bde95fb534a99b9722643fcd8d228b6a4529967bfe2f3dd4d57ec0f560d" Oct 31 01:16:37.235323 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:37.235290 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:37.385326 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:37.385283 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:37.911258 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:37.911228 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:16:56.036197 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:16:56.036118 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:04.279490 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:17:04.279454 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:17:22.092834 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:22.092799 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:17:22.125337 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:22.125300 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:24.286069 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:24.286035 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:24.842880 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:24.842849 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:34.279257 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:17:34.279215 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:17:34.858254 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:34.858216 4471 scope.go:122] "RemoveContainer" containerID="e078282db7c8a872c35e4e9b5d1ee2b4da044c18a8b4ac2c2b62313438a0c12a" Oct 31 01:17:34.870216 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:34.870190 4471 scope.go:122] "RemoveContainer" containerID="15c20d668037e44e10a62913390424887994fff486f4eaf0ad7dce2a35e0564a" Oct 31 01:17:37.112000 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.111956 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" podUID="45fe0be3-7c04-4752-8587-23a463343a55" containerName="main" containerID="cri-o://b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958" gracePeriod=45 Oct 31 01:17:37.355339 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.355312 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:17:37.439183 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439147 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439358 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439211 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439358 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439253 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439358 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439304 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439358 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439335 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\" (UniqueName: \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439608 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439368 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\") pod \"45fe0be3-7c04-4752-8587-23a463343a55\" (UID: \"45fe0be3-7c04-4752-8587-23a463343a55\") " Oct 31 01:17:37.439608 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439587 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:37.439728 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439656 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:37.439728 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439692 4471 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-model-cache\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.439823 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.439791 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:37.441808 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.441781 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:17:37.441808 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.441789 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:17:37.441952 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.441833 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr" pod "45fe0be3-7c04-4752-8587-23a463343a55" (UID: "45fe0be3-7c04-4752-8587-23a463343a55"). InnerVolumeSpecName "kube-api-access-brwrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:17:37.444724 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.444698 4471 generic.go:356] "Generic (PLEG): container finished" podID="45fe0be3-7c04-4752-8587-23a463343a55" containerID="b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958" exitCode=0 Oct 31 01:17:37.444818 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.444737 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" Oct 31 01:17:37.444818 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.444738 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" event={"ID":"45fe0be3-7c04-4752-8587-23a463343a55","Type":"ContainerDied","Data":"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958"} Oct 31 01:17:37.444818 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.444779 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns" event={"ID":"45fe0be3-7c04-4752-8587-23a463343a55","Type":"ContainerDied","Data":"ac7eeb77640bdbd59529dfd7672f0caa8791246d7c09c2363f6910a3713f4c79"} Oct 31 01:17:37.444818 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.444800 4471 scope.go:122] "RemoveContainer" containerID="b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958" Oct 31 01:17:37.459374 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.459358 4471 scope.go:122] "RemoveContainer" containerID="b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958" Oct 31 01:17:37.459682 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:17:37.459664 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958\": container with ID starting with b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958 not found: ID does not exist" containerID="b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958" Oct 31 01:17:37.459736 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.459692 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958"} err="failed to get container status \"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958\": rpc error: code = NotFound desc = could not find container \"b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958\": container with ID starting with b8e2fab672054825d9b778d040bcdbdbca787135f9fa8af0ccd069aa3cdaa958 not found: ID does not exist" Oct 31 01:17:37.540448 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.540417 4471 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-dshm\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.540448 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.540442 4471 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fe0be3-7c04-4752-8587-23a463343a55-tls-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.540448 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.540451 4471 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-tmp-dir\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.540711 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.540461 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwrr\" (UniqueName: \"kubernetes.io/projected/45fe0be3-7c04-4752-8587-23a463343a55-kube-api-access-brwrr\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.540711 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.540471 4471 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fe0be3-7c04-4752-8587-23a463343a55-home\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:17:37.569128 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.569099 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:17:37.571209 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.571183 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-df3d3673-kserve-59564cc55c-745ns"] Oct 31 01:17:37.586057 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:37.586029 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:17:39.207678 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:17:39.207646 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fe0be3-7c04-4752-8587-23a463343a55" path="/var/lib/kubelet/pods/45fe0be3-7c04-4752-8587-23a463343a55/volumes" Oct 31 01:18:04.279824 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:18:04.279788 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:18:12.045435 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.045398 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h"] Oct 31 01:18:12.091017 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.090984 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2c75h"] Oct 31 01:18:12.523385 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523349 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:12.523761 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523746 4471 cpu_manager.go:414] "containerMap: removing container" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" Oct 31 01:18:12.523815 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523763 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" Oct 31 01:18:12.523815 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523794 4471 cpu_manager.go:414] "containerMap: removing container" podUID="45fe0be3-7c04-4752-8587-23a463343a55" containerName="main" Oct 31 01:18:12.523815 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523800 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fe0be3-7c04-4752-8587-23a463343a55" containerName="main" Oct 31 01:18:12.523910 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523862 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="d394b94f-1433-4757-a0cc-ed05b0bfebd3" containerName="payload-pre-processing" Oct 31 01:18:12.523910 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.523875 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fe0be3-7c04-4752-8587-23a463343a55" containerName="main" Oct 31 01:18:12.527061 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.527045 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.600770 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.600734 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82r6z\" (UniqueName: \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.600963 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.600799 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.600963 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.600915 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.601136 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.601000 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.622003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.621976 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v4nxh\"" Oct 31 01:18:12.622003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.621993 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:18:12.622213 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.621976 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:18:12.623654 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.623634 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:12.701929 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.701891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.702105 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.701960 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82r6z\" (UniqueName: \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.702105 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.701995 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.702105 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.702019 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.702608 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.702586 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.704574 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.704553 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.704669 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.704574 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.759007 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.758977 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82r6z\" (UniqueName: \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\") pod \"maas-api-799fd7675f-hblng\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:12.844584 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:12.844495 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:13.208326 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:13.208292 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f791b43a-de66-4985-ac01-d7c72485c405" path="/var/lib/kubelet/pods/f791b43a-de66-4985-ac01-d7c72485c405/volumes" Oct 31 01:18:13.208733 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:13.208588 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:13.211435 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:18:13.211406 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d66609_6177_4378_b3ce_c8bab8a300c0.slice/crio-4f72fd92228516a377eef01ad9d12319641cb17880c6d149d550cf6bc15d15ee WatchSource:0}: Error finding container 4f72fd92228516a377eef01ad9d12319641cb17880c6d149d550cf6bc15d15ee: Status 404 returned error can't find the container with id 4f72fd92228516a377eef01ad9d12319641cb17880c6d149d550cf6bc15d15ee Oct 31 01:18:13.212725 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:13.212708 4471 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:18:13.622758 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:13.622648 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" event={"ID":"51d66609-6177-4378-b3ce-c8bab8a300c0","Type":"ContainerStarted","Data":"4f72fd92228516a377eef01ad9d12319641cb17880c6d149d550cf6bc15d15ee"} Oct 31 01:18:14.629386 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:14.629344 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" event={"ID":"51d66609-6177-4378-b3ce-c8bab8a300c0","Type":"ContainerStarted","Data":"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d"} Oct 31 01:18:14.629848 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:14.629408 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:14.741175 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:14.741121 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" podStartSLOduration=1.819308446 podStartE2EDuration="2.741106632s" totalImagesPullingTime="921.798186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:14.740030791 +0000 UTC m=+3821.665299773" watchObservedRunningTime="2026-10-31 01:18:14.741106632 +0000 UTC m=+3821.666375584" Oct 31 01:18:19.419613 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:19.419572 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:20.047978 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:20.047943 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:20.264301 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:20.264257 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:20.638576 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:20.638545 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:21.244014 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.243980 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25"] Oct 31 01:18:21.247783 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.247759 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.285919 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.285888 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz7h\" (UniqueName: \"kubernetes.io/projected/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-kube-api-access-qwz7h\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.286062 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.285932 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-openshift-service-ca\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.286062 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.285960 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-maas-api-tls\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.286062 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.286005 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-metrics-certs\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.309727 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.309697 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25"] Oct 31 01:18:21.386884 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.386847 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-openshift-service-ca\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.386884 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.386891 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-maas-api-tls\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.387152 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.386907 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-metrics-certs\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.387152 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.386979 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz7h\" (UniqueName: \"kubernetes.io/projected/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-kube-api-access-qwz7h\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.387568 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.387545 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-openshift-service-ca\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.389326 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.389294 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-maas-api-tls\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.389414 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.389356 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-metrics-certs\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.444350 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.444318 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz7h\" (UniqueName: \"kubernetes.io/projected/dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7-kube-api-access-qwz7h\") pod \"maas-api-5cbd79cdff-fvs25\" (UID: \"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7\") " pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.560589 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.560489 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:21.926445 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:21.926401 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25"] Oct 31 01:18:21.928701 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:18:21.928676 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6fbce8_b75a_4aca_baf2_019fd6e3e9b7.slice/crio-0a98a417ee1cb671e24707a9d29153e2e67ed87d5a59a3cf37a49ad5a78de193 WatchSource:0}: Error finding container 0a98a417ee1cb671e24707a9d29153e2e67ed87d5a59a3cf37a49ad5a78de193: Status 404 returned error can't find the container with id 0a98a417ee1cb671e24707a9d29153e2e67ed87d5a59a3cf37a49ad5a78de193 Oct 31 01:18:22.666616 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:22.666577 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" event={"ID":"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7","Type":"ContainerStarted","Data":"0a98a417ee1cb671e24707a9d29153e2e67ed87d5a59a3cf37a49ad5a78de193"} Oct 31 01:18:23.672640 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:23.672602 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" event={"ID":"dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7","Type":"ContainerStarted","Data":"ba321a6667b564e0b3d9ba7dc4a14d8a3371a16ae304a6c7de0df33ffb5458b0"} Oct 31 01:18:23.673033 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:23.672663 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:23.751209 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:23.751176 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:23.782619 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:23.782563 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" podStartSLOduration=1.7977692140000001 podStartE2EDuration="2.782550184s" totalImagesPullingTime="984.78097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:23.782121135 +0000 UTC m=+3830.707390087" watchObservedRunningTime="2026-10-31 01:18:23.782550184 +0000 UTC m=+3830.707819132" Oct 31 01:18:23.927138 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:23.927061 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:24.141890 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:24.141846 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:26.941660 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:26.941626 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:27.114455 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:27.114422 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:28.876386 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:28.876354 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:29.047186 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:29.047157 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:29.197051 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:29.197016 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:29.682073 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:29.682045 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cbd79cdff-fvs25" Oct 31 01:18:29.852665 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:29.852636 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:29.852867 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:29.852841 4471 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" podUID="51d66609-6177-4378-b3ce-c8bab8a300c0" containerName="maas-api" containerID="cri-o://e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d" gracePeriod=30 Oct 31 01:18:30.101276 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.101249 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:30.171588 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.171558 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\") pod \"51d66609-6177-4378-b3ce-c8bab8a300c0\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " Oct 31 01:18:30.171739 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.171675 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\") pod \"51d66609-6177-4378-b3ce-c8bab8a300c0\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " Oct 31 01:18:30.171739 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.171716 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\") pod \"51d66609-6177-4378-b3ce-c8bab8a300c0\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " Oct 31 01:18:30.171832 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.171786 4471 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\" (UniqueName: \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\") pod \"51d66609-6177-4378-b3ce-c8bab8a300c0\" (UID: \"51d66609-6177-4378-b3ce-c8bab8a300c0\") " Oct 31 01:18:30.171951 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.171930 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca" pod "51d66609-6177-4378-b3ce-c8bab8a300c0" (UID: "51d66609-6177-4378-b3ce-c8bab8a300c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:30.172111 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.172090 4471 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d66609-6177-4378-b3ce-c8bab8a300c0-openshift-service-ca\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:30.173792 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.173755 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs" pod "51d66609-6177-4378-b3ce-c8bab8a300c0" (UID: "51d66609-6177-4378-b3ce-c8bab8a300c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:30.173905 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.173810 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls" pod "51d66609-6177-4378-b3ce-c8bab8a300c0" (UID: "51d66609-6177-4378-b3ce-c8bab8a300c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:30.173905 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.173835 4471 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z" pod "51d66609-6177-4378-b3ce-c8bab8a300c0" (UID: "51d66609-6177-4378-b3ce-c8bab8a300c0"). InnerVolumeSpecName "kube-api-access-82r6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:30.273500 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.273433 4471 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-metrics-certs\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:30.273500 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.273459 4471 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d66609-6177-4378-b3ce-c8bab8a300c0-maas-api-tls\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:30.273500 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.273469 4471 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82r6z\" (UniqueName: \"kubernetes.io/projected/51d66609-6177-4378-b3ce-c8bab8a300c0-kube-api-access-82r6z\") on node \"ip-10-0-134-11.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:30.705688 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.705653 4471 generic.go:356] "Generic (PLEG): container finished" podID="51d66609-6177-4378-b3ce-c8bab8a300c0" containerID="e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d" exitCode=0 Oct 31 01:18:30.705866 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.705697 4471 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" Oct 31 01:18:30.705866 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.705738 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" event={"ID":"51d66609-6177-4378-b3ce-c8bab8a300c0","Type":"ContainerDied","Data":"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d"} Oct 31 01:18:30.705866 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.705788 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-799fd7675f-hblng" event={"ID":"51d66609-6177-4378-b3ce-c8bab8a300c0","Type":"ContainerDied","Data":"4f72fd92228516a377eef01ad9d12319641cb17880c6d149d550cf6bc15d15ee"} Oct 31 01:18:30.705866 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.705812 4471 scope.go:122] "RemoveContainer" containerID="e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d" Oct 31 01:18:30.715192 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.715174 4471 scope.go:122] "RemoveContainer" containerID="e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d" Oct 31 01:18:30.715426 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:18:30.715408 4471 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d\": container with ID starting with e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d not found: ID does not exist" containerID="e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d" Oct 31 01:18:30.715467 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.715437 4471 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d"} err="failed to get container status \"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d\": rpc error: code = NotFound desc = could not find container \"e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d\": container with ID starting with e972fda50797d3359ba21b30472c9d6f9f1820e0ee18bacad26da7d0d147108d not found: ID does not exist" Oct 31 01:18:30.857548 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.857506 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:30.903515 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:30.903481 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-799fd7675f-hblng"] Oct 31 01:18:31.207350 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:31.207305 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d66609-6177-4378-b3ce-c8bab8a300c0" path="/var/lib/kubelet/pods/51d66609-6177-4378-b3ce-c8bab8a300c0/volumes" Oct 31 01:18:34.279422 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:18:34.279386 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:18:34.397308 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:34.397275 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:37.914786 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:37.914749 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:38.305911 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:38.305821 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:39.509354 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:39.509317 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:44.599731 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:44.599699 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:50.088913 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:50.088873 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:53.464959 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:53.464917 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:53.611952 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:53.611915 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:18:55.638725 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:18:55.638687 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:00.785050 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:00.785015 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:04.279144 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:19:04.279107 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:19:05.904941 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:05.904907 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:11.012966 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:11.012929 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:12.109903 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:12.109869 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:12.257482 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:12.257445 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:15.983039 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:15.982997 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:20.454646 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:20.454609 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:22.046502 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:22.046466 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:22.346912 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:22.346833 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:26.559457 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:26.559421 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:27.877239 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:27.877182 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:28.516121 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:28.516084 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:28.711080 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:28.711047 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:28.855021 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:28.854936 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:32.700401 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:32.700362 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:32.853874 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:32.853834 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:33.002815 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:33.002733 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:34.278975 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:19:34.278941 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:19:34.920115 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:34.920080 4471 scope.go:122] "RemoveContainer" containerID="0760197229b35d8389f985f7dbd1c6b3025b294f949fff8e237ed18372e6f635" Oct 31 01:19:35.871766 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:35.871735 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:36.034968 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:36.034932 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:38.270242 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:38.270208 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:39.615560 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:39.615510 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:42.258926 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:42.258885 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:44.592334 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:44.592255 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:49.704239 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:49.704198 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:54.810518 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:54.810480 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.549267 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:19:57.549234 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:00.282886 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:00.282834 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:04.279603 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:20:04.279563 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:20:05.497007 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:05.496968 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.606158 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:10.606118 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:12.045439 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:12.045398 4471 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx"] Oct 31 01:20:12.091384 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:12.091349 4471 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f6ff4c-8sxlx"] Oct 31 01:20:13.207133 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:13.207100 4471 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c4fe7e-3639-412a-931c-5e10cbf9f72f" path="/var/lib/kubelet/pods/03c4fe7e-3639-412a-931c-5e10cbf9f72f/volumes" Oct 31 01:20:15.692918 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:15.692885 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:20.782843 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:20.782803 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.819670 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:25.819635 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:30.893995 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:30.893960 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:34.278830 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:20:34.278794 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:20:35.988003 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:35.987966 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:37.540266 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:37.540229 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:37.686876 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:37.686840 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:38.604721 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:38.604686 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:41.091242 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:41.091207 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:46.197404 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:46.197361 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:51.303414 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:51.303380 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:56.387334 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:20:56.387296 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:01.503337 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:01.503302 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.279402 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:21:04.279367 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:21:06.592116 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:06.592086 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.790904 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:11.790867 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:16.894342 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:16.894256 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:22.428619 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:22.428562 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:27.604920 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:27.604887 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.170029 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:30.169993 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:30.316856 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:30.316818 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:34.279140 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:21:34.279101 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:21:34.946374 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:34.946340 4471 scope.go:122] "RemoveContainer" containerID="2d1aac9c3be060d0dbffbab777fb9af7e6babc91bd47923f3e7272fd4b8f4d31" Oct 31 01:21:34.954413 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:34.954387 4471 scope.go:122] "RemoveContainer" containerID="3e665973423cf4adde81fb7d5df7afbecf3c7661ac436e0a19a9fbefe553fd61" Oct 31 01:21:37.154206 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:37.154172 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:38.773842 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:38.773811 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:38.960067 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:38.960024 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:39.107800 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:39.107717 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:39.331944 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:39.331896 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:39.878287 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:39.878255 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:45.858172 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:45.858138 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:46.009851 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:46.009817 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:46.218640 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:21:46.218602 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.278674 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:22:04.278634 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:22:34.296714 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:22:34.296662 4471 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-cdzdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:22:34.297375 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:22:34.297331 4471 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-cdzdr" podUID="668ca558-2510-451e-9fb4-8c261ed2933d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:22:34.345217 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:22:34.345167 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:22:39.310400 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:22:39.310366 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:41.232498 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:22:41.232466 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:50.640194 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:22:50.640100 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:04.280005 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:23:04.279966 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:23:11.204821 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:11.204792 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bs7tf_80847ae4-029e-42e4-8ed2-648f67e4242e/manager/0.log" Oct 31 01:23:11.597698 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:11.597616 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mfmng_b270816d-d116-40b5-9d03-955a269225bd/manager/0.log" Oct 31 01:23:14.180237 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:14.180199 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cbd79cdff-fvs25_dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7/maas-api/0.log" Oct 31 01:23:15.175247 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:15.175196 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qq8l9_cde4498f-bec4-4e51-b49d-b5161e82d106/postgres/0.log" Oct 31 01:23:17.657394 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:17.657364 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zxthq_95f5fe9d-cd0b-492c-ab83-a2c90700f662/kuadrant-console-plugin/0.log" Oct 31 01:23:19.851539 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:19.851505 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d4777458-v9hv8_d7731d80-c8a6-4db2-a068-d89a39303720/kube-auth-proxy/0.log" Oct 31 01:23:20.549746 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:20.549714 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-cdzdr_668ca558-2510-451e-9fb4-8c261ed2933d/istio-proxy/0.log" Oct 31 01:23:21.130086 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:21.130050 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-h4vr2_f5964ec3-db73-48d0-aba5-6ba343813cc1/istio-proxy/0.log" Oct 31 01:23:21.651353 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:21.651318 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-q4c82_ebfdd964-00a7-4e31-973d-a35e80995c06/istio-proxy/0.log" Oct 31 01:23:22.927118 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:22.927092 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj_6c2bbcde-ca2d-493b-8c69-8a81711e5faf/istio-proxy/0.log" Oct 31 01:23:34.280503 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:23:34.280463 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:23:37.774457 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:37.774429 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-bs7tf_80847ae4-029e-42e4-8ed2-648f67e4242e/manager/0.log" Oct 31 01:23:38.166018 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:38.165984 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mfmng_b270816d-d116-40b5-9d03-955a269225bd/manager/0.log" Oct 31 01:23:39.648459 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:39.648421 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:40.192254 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:40.192224 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cbd79cdff-fvs25_dc6fbce8-b75a-4aca-baf2-019fd6e3e9b7/maas-api/0.log" Oct 31 01:23:41.217953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:41.217922 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qq8l9_cde4498f-bec4-4e51-b49d-b5161e82d106/postgres/0.log" Oct 31 01:23:43.432593 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:43.432563 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zxthq_95f5fe9d-cd0b-492c-ab83-a2c90700f662/kuadrant-console-plugin/0.log" Oct 31 01:23:45.255958 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:45.255926 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66d4777458-v9hv8_d7731d80-c8a6-4db2-a068-d89a39303720/kube-auth-proxy/0.log" Oct 31 01:23:45.950545 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:45.950506 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-cdzdr_668ca558-2510-451e-9fb4-8c261ed2933d/istio-proxy/0.log" Oct 31 01:23:46.509006 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:46.508944 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-h4vr2_f5964ec3-db73-48d0-aba5-6ba343813cc1/istio-proxy/0.log" Oct 31 01:23:47.039008 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:47.038972 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-q4c82_ebfdd964-00a7-4e31-973d-a35e80995c06/istio-proxy/0.log" Oct 31 01:23:48.250010 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:23:48.249976 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-r4ncj_6c2bbcde-ca2d-493b-8c69-8a81711e5faf/istio-proxy/0.log" Oct 31 01:24:04.279650 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:24:04.279599 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:24:34.279740 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:24:34.279702 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:24:39.775252 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:24:39.775217 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:40.909953 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:24:40.909919 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.279823 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:25:04.279784 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:25:04.374483 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:04.374448 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.562359 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:04.562283 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:15.513510 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:15.513479 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t57nz_a0f381bf-9848-4fbe-8b12-acd6fb7aedb1/global-pull-secret-syncer/0.log" Oct 31 01:25:17.432282 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:17.432250 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" Oct 31 01:25:18.839797 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:18.839765 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-11.us-west-2.compute.internal_682248b34b0664951bb6ec422a0d79db/haproxy/0.log" Oct 31 01:25:24.120420 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.120387 4471 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px"] Oct 31 01:25:24.120838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.120753 4471 cpu_manager.go:414] "containerMap: removing container" podUID="51d66609-6177-4378-b3ce-c8bab8a300c0" containerName="maas-api" Oct 31 01:25:24.120838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.120763 4471 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d66609-6177-4378-b3ce-c8bab8a300c0" containerName="maas-api" Oct 31 01:25:24.120838 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.120837 4471 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d66609-6177-4378-b3ce-c8bab8a300c0" containerName="maas-api" Oct 31 01:25:24.124110 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.124092 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.176485 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.176459 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bw6xw\"/\"openshift-service-ca.crt\"" Oct 31 01:25:24.176655 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.176602 4471 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bw6xw\"/\"kube-root-ca.crt\"" Oct 31 01:25:24.228127 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.228096 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px"] Oct 31 01:25:24.235057 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.235024 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-podres\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.235196 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.235087 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j952q\" (UniqueName: \"kubernetes.io/projected/6430efba-717d-4268-b0dd-5d028151298b-kube-api-access-j952q\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.235196 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.235157 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-proc\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.235270 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.235205 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-sys\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.235270 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.235233 4471 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-lib-modules\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.268012 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.267978 4471 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bw6xw\"/\"default-dockercfg-vkhnz\"" Oct 31 01:25:24.336746 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336707 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-podres\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336753 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j952q\" (UniqueName: \"kubernetes.io/projected/6430efba-717d-4268-b0dd-5d028151298b-kube-api-access-j952q\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336778 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-proc\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336799 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-sys\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336846 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-podres\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336894 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-proc\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336849 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-sys\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.336933 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.336929 4471 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-lib-modules\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.337173 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.337010 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6430efba-717d-4268-b0dd-5d028151298b-lib-modules\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.398314 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.398283 4471 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j952q\" (UniqueName: \"kubernetes.io/projected/6430efba-717d-4268-b0dd-5d028151298b-kube-api-access-j952q\") pod \"perf-node-gather-daemonset-pw8px\" (UID: \"6430efba-717d-4268-b0dd-5d028151298b\") " pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.435913 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.435888 4471 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:24.787017 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.786948 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zxthq_95f5fe9d-cd0b-492c-ab83-a2c90700f662/kuadrant-console-plugin/0.log" Oct 31 01:25:24.803222 ip-10-0-134-11 kubenswrapper[4471]: W1031 01:25:24.803194 4471 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6430efba_717d_4268_b0dd_5d028151298b.slice/crio-cc74ebf62651af2246f77decddc9fec94537aaa65a6a37c6a29501e626ba6b9a WatchSource:0}: Error finding container cc74ebf62651af2246f77decddc9fec94537aaa65a6a37c6a29501e626ba6b9a: Status 404 returned error can't find the container with id cc74ebf62651af2246f77decddc9fec94537aaa65a6a37c6a29501e626ba6b9a Oct 31 01:25:24.805468 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:24.805443 4471 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px"] Oct 31 01:25:25.535819 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:25.535785 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" event={"ID":"6430efba-717d-4268-b0dd-5d028151298b","Type":"ContainerStarted","Data":"f0928e9006f506acd27ff77bc9787aa594365367a1f0cbda96df30939c1c023b"} Oct 31 01:25:25.536223 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:25.535834 4471 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" event={"ID":"6430efba-717d-4268-b0dd-5d028151298b","Type":"ContainerStarted","Data":"cc74ebf62651af2246f77decddc9fec94537aaa65a6a37c6a29501e626ba6b9a"} Oct 31 01:25:25.536223 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:25.535861 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:25.655448 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:25.655399 4471 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" podStartSLOduration=1.6553853429999998 podStartE2EDuration="1.655385343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:25.653678183 +0000 UTC m=+4252.578947134" watchObservedRunningTime="2026-10-31 01:25:25.655385343 +0000 UTC m=+4252.580654331" Oct 31 01:25:31.549337 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:31.549309 4471 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bw6xw/perf-node-gather-daemonset-pw8px" Oct 31 01:25:33.763854 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:33.763824 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-j7f6c_d4b2dd2a-fd57-4667-ad81-b205b7bacbcc/monitoring-plugin/0.log" Oct 31 01:25:34.280057 ip-10-0-134-11 kubenswrapper[4471]: E1031 01:25:34.280021 4471 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t25p7" containerName="extractor" Oct 31 01:25:35.459990 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:35.459960 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k9r7z_48004d0a-ef43-4245-8788-a0f823e3c8da/node-exporter/0.log" Oct 31 01:25:35.887680 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:35.887596 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k9r7z_48004d0a-ef43-4245-8788-a0f823e3c8da/kube-rbac-proxy/0.log" Oct 31 01:25:36.320807 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:36.320778 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k9r7z_48004d0a-ef43-4245-8788-a0f823e3c8da/init-textfile/0.log" Oct 31 01:25:39.396301 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:39.396272 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/prometheus/0.log" Oct 31 01:25:39.887992 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:39.887916 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/config-reloader/0.log" Oct 31 01:25:40.316956 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:40.316929 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/thanos-sidecar/0.log" Oct 31 01:25:40.340018 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:40.339991 4471 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4rxkn_37a308b8-65d6-4b6d-a1de-1c7af22ac808/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:40.750712 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:40.750681 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/kube-rbac-proxy-web/0.log" Oct 31 01:25:41.188588 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:41.188561 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/kube-rbac-proxy/0.log" Oct 31 01:25:41.617793 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:41.617720 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/kube-rbac-proxy-thanos/0.log" Oct 31 01:25:42.046718 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:42.046690 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b8fa41aa-7f57-41b1-b056-4bcfd23356c4/init-config-reloader/0.log" Oct 31 01:25:42.486699 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:42.486669 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rj944_4fd3ce17-9e54-4115-9e24-82ea6ac4551c/prometheus-operator/0.log" Oct 31 01:25:42.912765 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:42.912737 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rj944_4fd3ce17-9e54-4115-9e24-82ea6ac4551c/kube-rbac-proxy/0.log" Oct 31 01:25:43.343543 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:43.343426 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-srvsp_ee61e853-22b2-487f-96a2-8ce7aa360d06/prometheus-operator-admission-webhook/0.log" Oct 31 01:25:45.027155 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:45.027119 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/thanos-query/0.log" Oct 31 01:25:45.456071 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:45.456040 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/kube-rbac-proxy-web/0.log" Oct 31 01:25:45.883850 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:45.883785 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/kube-rbac-proxy/0.log" Oct 31 01:25:46.311763 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:46.311730 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/prom-label-proxy/0.log" Oct 31 01:25:46.740967 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:46.740936 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/kube-rbac-proxy-rules/0.log" Oct 31 01:25:47.171698 ip-10-0-134-11 kubenswrapper[4471]: I1031 01:25:47.171670 4471 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fb76784f8-nc45q_7afe3f17-55c3-4edd-bfce-ddf42f5230f5/kube-rbac-proxy-metrics/0.log"